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In  2001 ,  the  National  Nuclear  Security  Administration  of  the  U.S.  Department  of  Energy  in  conjunction 
with  the  national  security  laboratories  (i.e.,  Los  Alamos  National  Laboratory,  Lawrence  Livermore 
National  Laboratory  and  Sandia  National  Laboratories)  initiated  development  of  a  process  designated 
Quantification  of  Margins  and  Uncertainties  (QMU)  for  the  use  of  risk  assessment  methodologies  in  the 
certification  of  the  reliability  and  safety  of  the  nation’s  nuclear  weapons  stockpile.  This  presentation 
discusses  and  illustrates  the  conceptual  and  computational  basis  of  QMU  in  analyses  that  use 
computational  models  to  predict  the  behavior  of  complex  systems.  The  following  topics  are  considered: 
(i)  the  role  of  aleatory  and  epistemic  uncertainty  in  QMU,  (ii)  the  representation  of  uncertainty  with 
probability,  (iii)  the  probabilistic  representation  of  uncertainty  in  QMU  analyses  involving  only 
epistemic  uncertainty,  and  (iv)  the  probabilistic  representation  of  uncertainty  in  QMU  analyses 
involving  aleatory  and  epistemic  uncertainty. 
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1.  Introduction 

In  2001,  the  National  Nuclear  Security  Administration  (NNSA) 
of  the  U.S.  Department  of  Energy  (DOE)  in  conjunction  with  the 
national  security  laboratories  (i.e.,  Los  Alamos  National  Labora¬ 
tory,  Lawrence  Livermore  National  Laboratory  and  Sandia  National 
Laboratories)  initiated  development  of  a  process  designated  Quan¬ 
tification  of  Margins  and  Uncertainties  (QMU)  for  the  use  of  risk 
assessment  methodologies  in  the  certification  of  the  reliability  and 
safety  of  the  nation’s  nuclear  weapons  stockpile  [1-6],  Specifically, 
the  following  requirements  have  been  proposed  [7]: 

Design  agency  assessments  shall  incorporate  QMU  methodolo¬ 
gies  as  an  essential  part  of  the  framework  necessary  for  the 
evaluation  of  the  performance  of  warhead  and  warhead  com¬ 
ponents.  QMU  can  be  used  as  one  of  the  tools  for  identification 
and  prioritization  of  actions  required  for  a  component  or 
system.  Issues  that  require  immediate  attention  must  be  raised 
to  the  NNSA  Office  of  Stockpile  Assessments  and  Certification. 
The  design  agency  laboratories  shall  develop  site-appropriate 
QMU  implementation  plans.  (NNSA-1) 

Certification,  qualification,  and  significant  finding  investiga¬ 
tions  closure  plans  shall  include  QMU  methodologies  where 
applicable.  Results  of  assessments  using  QMU  shall  be  included 
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in  warhead  certification  documents,  component  qualification 
documents,  annual  assessment  reports  (AARs)  and  Significant 
Finding  Investigation  (SFI)  closure  documentation.  (NNSA-2) 

As  indicated  by  the  preceding  statements,  the  NNSA  intends 
for  QMU  to  be  an  integral  component  of  the  assessment  process 
for  the  nation’s  nuclear  weapons  stockpile.  However,  the  preced¬ 
ing  statements  give  no  indication  of  what  the  NNSA  envisions  as 
the  conceptual  and  computational  basis  for  QMU.  In  this  regard, 
some  additional  information  with  respect  to  the  NNSA’s  intent 
for  QMU  is  provided  by  the  following  definitions  supplied  in 
conjunction  with  the  preceding  statements  [7]: 

Quantification  of  Margins  and  Uncertainties  is  a  scientific 
methodology  that  identifies  relevant  nuclear-warhead  para¬ 
meters  and  quantifies,  using  available  experimental  and  com¬ 
putational  tools,  the  margin  of  that  parameter  relative  to  its 
failure  point  and  the  uncertainties  associated  with  the  para¬ 
meter  and  the  failure  point.  An  assessment  of  the  relationship 
between  the  margin  and  uncertainties  facilitates  stockpile 
management  decisions,  resource  allocation  prioritization,  and 
informed  judgments  on  the  safety,  reliability  and  performance 
of  nuclear  warheads.  (NNSA-3) 

Uncertainty  is  a  best  estimate  of  the  range  of  a  particular  metric 
which  may  derive  from  one  or  two  broad  sources.  Uncertainties 
that  reflect  a  lack  of  knowledge  about  the  appropriate  value 
to  use  for  a  quantity  that  is  assumed  to  have  (missing  modifier: 
a  fixed?)  value  in  the  context  of  a  particular  analysis  are 
termed  epistemic.  Uncertainties  that  arise  from  an  inherent 
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randomness  in  the  behavior  of  the  system  under  study  are 

termed  aleatoric.  (NNSA-4) 

Although  designated  as  definitions,  the  statements  in  Quotes 
(NNSA-3)  and  (NNSA-4)  are  at  a  high  level  and  lack  specifics.  For 
example,  QMU  is  defined  in  Quote  (NNSA-3)  as  a  “scientific 
methodology”  but  no  details  are  given  with  respect  to  what  the 
conceptual  basis  and  resultant  computational  implementation  of 
this  methodology  should  be.  This  lack  of  specificity  is  consistent 
with  the  requirement  in  Quote  (NNSA-1)  that  “site-appropriate 
QMU  implementation  plans”  should  be  developed  and  has  the 
advantage  of  allowing  QMU  to  be  developed  and  implemented  in 
manners  appropriate  for  specific  analysis  contexts.  However,  this 
lack  of  specificity  does  not  exempt  individual  analyses  from  a 
requirement  to  clearly  define  their  conceptual  basis  and  asso¬ 
ciated  computational  implementation.  Such  definitions  are  essen¬ 
tial  if  a  specific  use  of  QMU  is  to  be  considered  a  “scientific 
methodology.” 

What  is  unambiguous  from  Quote  (NNSA-3)  is  that  the  appro¬ 
priate  treatment  of  uncertainty  is  to  be  an  integral  part  of  any 
implementation  of  QMU.  The  nature  of  uncertainty  and  the 
division  of  uncertainty  into  epistemic  and  aleatory  components 
is  elaborated  on  in  Quote  (NNSA-4).  This  is  an  important  distinc¬ 
tion  that  can  have  significant  effects  on  the  conceptual  basis  and 
computational  design  of  an  analysis  and  also  on  the  interpretation 
of  the  results  of  an  analysis. 

When  viewed  at  a  high  level,  the  application  of  QMU  can  be 
divided  into  two  distinct  cases:  (i)  comparison  of  experimental 
results  against  a  requirement  without  the  use  of  a  mathematical 
model  to  transform  the  experimental  results,  and  (ii)  comparison 
of  predictions  from  a  mathematical  model  against  a  requirement. 
This  presentation  is  restricted  to  the  second  case,  and  as  a  result, 
the  presented  concepts,  computational  procedures  and  discus¬ 
sions  should  be  viewed  in  the  context  of  comparing  model 
(i.e.,  computer)  predictions  with  a  requirement.  In  particular,  the 
strictly  statistical  issues  associated  with  the  direct  comparison  of 
experimental  results  with  a  requirement  are  not  considered. 

Although  the  descriptor  “risk  assessment”  does  not  appear  in 
Quotes  (NNSA-1  )-(NNSA-4),  the  QMU  process  being  described  in 
these  quotes  is  clearly  a  form  of  risk  assessment  in  that  it  involves 
the  determination  of  consequences  (i.e.,  analysis  outcomes  and 
associated  margins),  likelihoods  (i.e.,  the  effects  of  aleatory 
uncertainties),  and  state  of  knowledge  uncertainties  (i.e.,  episte¬ 
mic  uncertainties).  Such  determinations  are  the  essence  of  a  risk 
assessment.  As  a  result,  the  NNSA’s  mandate  for  QMU  is  a 
continuation  of  the  extensive  and  ongoing  use  of  risk  assessment 
in  many  different  areas.  As  indicated  in  the  following  three 
paragraphs,  there  is  an  extensive  body  of  prior  studies  and 
techniques  that  are  relevant  to  the  NNSA’s  mandated  use  of  QMU. 

Risk  assessment  for  complex  systems  has  a  long  history  and 
many  examples  relevant  to  QMU  exist,  including  (i)  early  studies 
of  missile  reliability  (Ref.  [8],  Section  3.2),  (ii)  the  U.S.  Nuclear 
Regulatory  Commission’s  (NRC’s)  assessment  of  the  risk  from 
commercial  nuclear  power  plants,  which  is  known  as  WASH- 1400 
after  a  report  number  [9],  (iii)  the  NRC’s  reassessment  of  the  risk 
from  commercial  nuclear  power  plants,  which  is  known  as 
NUREG-1150  after  a  report  number  [10,11],  (iv)  the  NRC’s  study 
of  margins  in  reactor  safety  [12-18],  (v)  the  NRC’s  analysis  of  the 
LaSalle  Nuclear  Power  Station  as  part  of  its  Risk  Methods 
Investigation  and  Evaluation  Program  [19],  (vi)  the  DOE’s  perfor¬ 
mance  assessment  for  the  Waste  Isolation  Pilot  Plant  (WIPP)  in 
support  of  a  successful  Compliance  Certification  Application  to 
the  U.S.  Environmental  Protection  Agency  (EPA)  [20,21],  and  (vii) 
the  DOE’s  performance  assessment  for  the  proposed  repository 
for  high-level  radioactive  waste  at  Yucca  Mountain,  Nevada, 
carried  out  in  support  of  a  licensing  application  to  the  NRC  [22], 


The  NRC’s  WASH-1400  analysis  is  rightfully  considered  to  be 
the  seminal  study  in  the  analysis  of  complex  systems.  After  its 
completion,  the  NRC  commissioned  a  review  of  the  WASH-1400 
analysis  known  as  the  Lewis  Committee  Report  after  the  chair¬ 
man  of  the  review  committee  [23],  This  review  was  highly 
complimentary  with  respect  to  the  overall  WASH-1400  analysis 
but  noted  that  the  analysis  had  inadequately  represented  the 
(epistemic)  uncertainty  in  its  results.  This  led  to  an  extensive 
interest  in  the  appropriate  incorporation  of  epistemic  uncertainty 
into  analyses  for  complex  systems  and  significantly  influenced 
the  NRC’s  program  to  develop  a  risk  assessment  methodology  to 
assess  the  geologic  disposal  of  high-level  radioactive  waste 
[24-26],  the  NRC’s  development  of  the  MELCOR  code  system  for 
the  analysis  of  nuclear  reactor  accidents  [27-29],  and  the  design 
and  implementation  of  the  analyses  indicated  in  (iii) -(vii)  above. 
Similarly  to  the  analyses  in  (iii)-(vii),  NNSA’s  mandate  for  QMU 
is  effectively  one  more  descendent  of  the  WASH-1400  analyses 
and  the  associated  Lewis  Committee  Report. 

Additional  information  on  the  development  of  risk  assessment 
methods  for  complex  systems  is  available  in  the  excellent  review 
by  Rechard  [8].  The  recent  review  by  Zio  is  also  a  valuable  source 
of  background  and  perspectives  on  risk  and  reliability  analysis  for 
complex  systems  [30],  Further,  the  book  by  Bernstein  is  highly 
recommended  for  a  broader  perspective  on  the  evolution  of  the 
ideas  underlying  the  assessment  of  risk  [31], 

The  QMU  process  also  quite  naturally  falls  into  a  broad  area  of 
study  known  as  uncertainty  and  sensitivity  analysis,  where 
uncertainty  analysis  refers  to  the  determination  of  the  uncer¬ 
tainty  in  analysis  results  that  derives  from  uncertainty  in  analysis 
inputs  and  sensitivity  analysis  refers  to  the  determination  of  the 
contributions  of  the  uncertainty  in  individual  analysis  inputs  to 
the  uncertainty  in  analysis  results.  The  uncertainty  being  referred 
to  in  the  preceding  sentence  is  usually  of  an  epistemic  nature. 
Clearly,  uncertainty  analysis  is  a  fundamental  component  of 
QMU;  indeed,  when  viewed  broadly,  QMU  is  simply  a  call  for 
uncertainty  analyses  focused  on  margins  (i.e.,  differences 
between  required  performance  and  obtainable  performance) 
associated  with  the  assessment  of  nuclear  weapon  reliability 
and  performance.  However,  sensitivity  analysis  is  also  a  funda¬ 
mental  part  of  QMU  as  indicated  by  reference  to  “identification 
and  prioritization  of  actions”  in  Quote  (NNSA-1)  and  “assessment 
of  the  relationship  between  margin  and  uncertainties”  in  Quote 
(NNSA-3).  Specifically,  the  indicated  actions  require  a  sensitivity 
analysis  to  determine  the  effects  of  the  uncertainty  in  individual 
analysis  inputs  on  the  uncertainty  in  analysis  results  of  interest 
(e.g.,  margins). 

As  a  result  of  its  fundamental  importance  in  analyses  of 
complex  systems,  a  number  of  approaches  to  uncertainty  and 
sensitivity  analysis  have  been  developed,  including  differential 
analysis  [32-37],  response  surface  methodology  [38-44],  Monte 
Carlo  analysis  [29,45-56],  and  variance  decomposition  proce¬ 
dures  [57-61].  Overviews  of  these  approaches  are  available  in 
several  reviews  [62-71],  Of  the  indicated  approaches  to  uncer¬ 
tainty  and  sensitivity  analysis,  sampling-based  (i.e.,  Monte  Carlo) 
approaches  are  likely  to  be  the  most  generally  useful  in  QMU 
analyses.  The  review  by  Helton  et  al.  [56]  provides  an  introduc¬ 
tion  to  analyses  of  this  type  and  also  background  and  additional 
references  for  many  of  the  ideas  introduced  in  this  presentation. 

A  number  of  presentations  discussing  the  QMU  process  are 
available  [2,72-77],  Of  particular  importance  is  the  recently 
published  National  Academy  of  Science/National  Research  Coun¬ 
cil  (NAS/NRC)  report,  which  provides  an  overview  of,  and  a  broad 
perspective  on,  QMU  at  the  national  security  laboratories  [77], 
However,  these  presentations  tend  to  be  written  at  a  high  level 
and,  as  a  result,  lack  detail  on  the  conceptual  basis  and  computa¬ 
tional  organization  that  must  underlie  a  real  QMU  analysis  if  that 
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analysis  is  to  be  a  manifestation  of  a  “scientific  methodology”  as 
indicated  in  Quote  (NNSA-3). 

The  purpose  of  this  presentation  is  to  describe  the  conceptual 
basis  and  computational  organization  of  QMU  analyses  that  use 
models  to  produce  results  that  are  then  compared  with  require¬ 
ments.  The  basic  idea  is  that  a  QMU  analysis  must  start  with  a 
clear  understanding  of  the  conceptual  (i.e.,  mathematical)  model 
used  to  represent  uncertainty.  In  turn,  this  model  leads  to  (i)  the 
manner  in  which  the  uncertainty  in  individual  analysis  inputs  is 
characterized,  (ii)  the  procedures  that  are  used  to  propagate 
uncertainty  through  the  analysis,  (iii)  the  procedures  that  are 
available  for  sensitivity  analysis,  and  (iv)  the  interpretations  and 
representations  that  are  available  for  analysis  results  of  interest 
(e.g.,  margins).  It  is  important  to  recognize  that  in  most  real 
analyses  there  will  probably  be  many  results  of  interest  in 
addition  to  a  single  margin  that  is  the  outcome  of  comparing  a 
single  calculated  result  with  a  single  requirement. 

The  presentation  is  organized  as  follows.  First,  the  important 
concepts  of  aleatory  and  epistemic  uncertainty  are  discussed 
(Section  2).  Next,  the  use  of  probability  in  the  representation  of 
uncertainty  is  described  and  two  example  problems  are  intro¬ 
duced  that  will  be  used  to  illustrate  different  potential  QMU 
analyses  (Section  3).  Specifically,  the  first  example  problem 
involves  only  epistemic  uncertainty  and  is  used  to  illustrate 
QMU  analyses  that  involve  only  epistemic  uncertainty  (Section  4). 
The  second  example  problem  involves  both  aleatory  and  episte¬ 
mic  uncertainty  and  is  used  to  illustrate  QMU  analyses  that 
involve  both  aleatory  and  epistemic  uncertainty  (Section  5).  The 
presentation  then  ends  with  a  summary  discussion  (Section  6). 
The  content  of  this  presentation  is  adapted  from  Sections  1-5  and 
11  of  Ref.  [78], 

For  added  perspective,  additional  information  on  the  implemen¬ 
tation  and  interpretation  of  analyses  involving  QMU  is  provided 
in  two  companion  presentations  [79,80],  Specifically,  Ref.  [79] 
describes  the  presence  of  QMU-type  margin  analyses  in  several  real, 
complex  and  computationally  demanding  analyses  that  involve  a 
separation  of  aleatory  and  epistemic  uncertainty,  and  Ref.  [80] 
describes  alternative  mathematical  structures  for  the  representation 
of  uncertainty  (i.e.,  interval  analysis,  possibility  theory,  evidence 
theory,  and  probability  theory)  and  then  illustrates  the  use  of  these 
structures  with  notional  QMU  analyses  involving  only  epistemic 
uncertainty  and  both  aleatory  and  epistemic  uncertainty. 


2.  Types  of  uncertainty 

In  the  design  and  implementation  of  analyses  for  complex 
systems,  it  is  useful  to  distinguish  between  two  types  of  uncer¬ 
tainty:  aleatory  uncertainty  and  epistemic  uncertainty  [81-93]. 
The  importance  of  this  distinction  is  recognized  by  the  NNSA  in 
Quote  (NNSA-4)  and  also  emphasized  in  the  NAS/NRC  report  on 
QMU  (Finding  1-3,  pp.  22-23,  Ref.  [77]). 

Aleatory  uncertainty  arises  from  an  inherent  randomness  in 
the  properties  or  behavior  of  the  system  under  study.  For 
example,  the  weather  conditions  at  the  time  of  a  reactor  accident 
are  inherently  random  with  respect  to  our  ability  to  predict  the 
future.  Other  examples  include  the  variability  in  the  properties  of 
a  population  of  weapon  components  and  the  variability  in  the 
possible  future  environmental  conditions  that  a  weapon  compo¬ 
nent  could  be  exposed  to.  Alternative  designations  for  aleatory 
uncertainty  include  variability,  stochastic,  irreducible  and  type  A. 

Epistemic  uncertainty  derives  from  a  lack  of  knowledge  about 
the  appropriate  value  to  use  for  a  quantity  that  is  assumed  to  have 
a  fixed  value  in  the  context  of  a  particular  analysis.  For  example, 
the  pressure  at  which  a  given  reactor  containment  would  fail  for  a 
specified  set  of  pressurization  conditions  is  fixed  but  not  amenable 


to  being  unambiguously  defined.  Other  examples  include  mini¬ 
mum  voltage  required  for  the  operation  of  a  system  and  the 
maximum  temperature  that  a  system  can  withstand  before  failing. 
Alternative  designations  for  epistemic  uncertainty  include  state  of 
knowledge,  subjective,  reducible  and  type  B. 

The  appropriate  separation  of  aleatory  and  epistemic  uncer¬ 
tainty  is  an  important  component  of  the  design  and  computational 
implementation  of  an  analysis  of  a  complex  system  and  also  of  the 
decisions  that  are  made  on  the  basis  of  this  analysis.  This  point  can 
be  made  with  a  simple  notional  example.  Suppose  an  analysis 
concludes  that  the  probability  of  a  particular  component  failing  to 
operate  correctly  is  0.01.  Without  the  specification  of  additional 
information,  there  are  two  possible  interpretations  to  the  indi¬ 
cated  probability.  The  first  interpretation,  which  is  inherently 
aleatoric,  is  that  1  in  every  100  components  of  this  type  will  fail 
to  operate  properly;  or,  put  another  way,  there  is  a  probability  of 
0.99  that  a  randomly  selected  component  will  operate  properly 
and  a  probability  of  0.01  that  a  randomly  selected  component  will 
not  operate  properly.  The  second  interpretation,  which  is  inher¬ 
ently  epistemic,  is  that  there  is  a  probability  of  0.99  that  all 
components  of  this  type  will  operate  properly  and  a  probability  of 
0.01  that  no  components  of  this  type  will  operate  properly. 
Clearly,  the  implications  of  the  two  interpretations  of  the  indicated 
probability  are  very  different,  and  as  a  consequence,  any  resultant 
decisions  about  the  system  under  study  can  also  be  expected  to  be 
very  different. 

The  analysis  of  a  complex  system  typically  involves  answering 
the  following  three  questions  about  the  system: 

What  can  happen?  (Ql) 

How  likely  is  it  to  happen?  (Q2) 

What  are  the  consequences  if  it  happens?  (Q3) 

and  one  additional  question  about  the  analysis  itself: 

How  much  confidence  exists  in  the  answers  to  the  first  three 

questions?  (Q4) 

The  answers  to  Questions  Ql  and  Q2  involve  the  characterization 
of  aleatory  uncertainty,  and  the  answer  to  Question  Q4  involves 
the  characterization  of  epistemic  uncertainty.  The  answer  to 
Question  Q3  typically  involves  numerical  modeling  of  the  system 
conditional  on  specific  realizations  of  aleatory  and  epistemic 
uncertainty.  The  posing  and  answering  of  Questions  Ql  -  Q3  gives 
rise  to  what  is  often  referred  to  as  the  Kaplan-Garrick  ordered 
triple  representation  for  risk  [92],  which  is  discussed  in  more 
detail  in  Section  3.7. 

The  use  of  probability  to  characterize  both  aleatory  uncertainty 
and  epistemic  uncertainty  is  described  and  illustrated  in  Sections 
3-5  and  can  be  traced  back  to  at  least  the  beginning  of  the  formal 
development  of  probability  theory  in  the  late  seventeenth  century 
[31,94,95].  However,  as  discussed  and  illustrated  in  Ref.  [80], 
several  alternative  mathematical  structures  for  the  representation 
of  epistemic  uncertainty  have  been  developed  in  the  last  several 
decades.  It  is  possible  that  some  of  these  alternative  structures 
may  be  more  appropriate  than  probability  in  certain  contexts  for 
the  representation  of  epistemic  uncertainty. 


3.  Representation  of  uncertainty  with  probability 

The  following  topics  related  to  the  representation  of  uncer¬ 
tainty  with  probability  are  now  introduced:  probability  spaces, 
cumulative  distribution  functions,  and  complementary  cumula¬ 
tive  distribution  functions  (Section  3.1),  the  basic  entities  that 
underlie  an  analysis  that  involves  a  representation  of  uncertainty 
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(Section  3.2),  analysis  in  the  presence  of  only  epistemic  uncer¬ 
tainty  (Section  3.3),  an  example  analysis  in  the  presence  of  only 
epistemic  uncertainty  (Section  3.4),  analysis  in  the  presence  of 
aleatory  and  epistemic  uncertainty  (Section  3.5),  an  example 
analysis  in  the  presence  of  aleatory  and  epistemic  uncertainty 
(Section  3.6),  the  Kaplan-Garrick  ordered  triple  representation  for 
risk  (Section  3.7),  verification  and  validation  (Section  3.8),  and  an 
admonition  about  the  importance  of  a  clear  specification  of 
concepts  in  the  representation  of  uncertainty  (Section  3.9). 

The  NAS/NRC  report  on  QMU  emphasizes  the  importance  of 
formal  uncertainty  quantification  (Finding  1-2,  p.  20,  Ref.  [77]). 
The  concepts  and  mathematical  structures  introduced  in  this 
section  are  fundamental  to  such  quantification. 


3.1.  Probability  spaces,  cumulative  distribution  functions  and 
complementary  cumulative  distribution  functions 

Probability  provides  the  mathematical  structure  traditionally 
used  to  represent  both  aleatory  uncertainty  and  epistemic  uncer¬ 
tainty  [81,83,86,89,90],  Formally,  a  probabilistic  characterization 
of  the  uncertainty  associated  with  a  quantity  x  is  provided  by  a 
probability  space  (X,  X,  px),  where  (i)  X  is  the  set  of  all  possible 
values  for  x,  (ii)  X  is  a  suitably  restricted  set  of  subsets  of  X  for 
which  probability  is  defined,  and  (iii)  px  is  a  function  that  defines 
probability  for  individual  elements  of  X  (i.e.,  if  U  e  X,  then  px(U) 
is  the  probability  of  U)  (Section  4.4,  Ref.  [96]).  Additional  discus¬ 
sion  of  probability  spaces  is  provided  in  Section  2.4  of  Ref.  [80]. 

In  practice,  a  probability  space  (X,  X,  px)  is  often  represented 
by  a  density  function  dx(x),  where 

px(U)  =  jdx(x)dU  (3.1) 

for  UeX.  Integrals  of  the  form  appearing  in  Eq.  (3.1)  are  usually 
taken  to  be  Lebesgue  integrals  in  formal  developments  of  prob¬ 
ability  theory  (e.g.,  [96,97]).  However,  for  the  purposes  of  this 
presentation,  all  presented  integrals  can  be  intuitively  thought  of 
as  corresponding  to  the  Riemann  integral  of  elementary  calculus. 
In  computational  practice,  high-dimensional  integrals  involving 
probability  spaces  are  usually  evaluated  with  sampling-based 
(i.e.,  Monte  Carlo)  procedures. 

When  x  corresponds  to  a  scalar  x  rather  than  a  vector,  a 
probability  space  (X,  X,  px)  can  be  summarized  with  a  cumulative 
distribution  function  (CDF)  or  a  complementary  cumulative 
distribution  function  (CCDF).  Specifically,  the  CDF  and  CCDF  for 
x  are  defined  by  plots  of  the  points 

[x,px(Ux)}  and  [x,px(Ucx)\,  (3.2) 

respectively,  for  x  e  X,  where 

Ux  =  {X:XeX  and  x  < x}, 

px(ftx)  =  probability  of  Ux  (i.e.,  of  a  value  x  <  x) 

=  £  dx(x)dx  =  jjfx(x)dx(x)  dx, 


Px(Ux)  =  probability  of  Ux  (i.e.,  of  a  value  x  >  x) 


=  jTdx(x)dx  =  j^dx(x)dx(x)dx, 


if  x  <  x 
otherwise 


and 


<5x(x)  =  l-c5v(x)  =  I  l  lf*>X. 

-*  [0  otherwise. 

Further,  it  is  usually  assumed  for  plotting  purposes  that 
(i)  px(Ux)  =  1  and  px(W£)  =  0  for  x  >  sup(A')  and  (ii)  px(Ux)  =  0 
and  Px(Ux)  =  1  for  x  <  \nUX). 

The  results  of  risk  assessments  are  often  summarized  with 
CCDFs  because  CCDFs  provide  an  answer  to  the  question  “How 
likely  is  it  to  be  this  large  or  larger?”,  which  is  typically  the  type 
of  question  that  risk  assessments  are  intended  to  answer.  In 
contrast,  CDFs  answer  the  question  “How  likely  is  it  to  be  this 
small  or  smaller?”,  which  is  likely  to  be  the  question  of  primary 
interest  in  a  margin  analysis. 

As  an  example,  the  CDF  and  CCDF  for  x  with  a  loguniform 
distribution  on  [2,10]  is  presented  in  Fig.  3.1.  For  this  example, 
dx(x)=l/[xln(10/2)]  =  l/[xln(5)],  2<x<  10,  (3.3) 

is  the  corresponding  density  function,  and  the  probabilities  px(Ux) 
and  px(Ux)  that  define  the  CDF  and  CCDF  are  given  by 

Px(Ux)  =  f2  {l/[x ln(5)]} dx  =  ln(x/2)/ln(5)  (3.4) 

px(Ucx)  =  j™{  l/[xln(5)]}dx  =  ln(10/x)/ln(5),  (3.5) 

respectively,  for  2<x<10.  In  practice,  most  probability  spaces 
and  their  associated  density  functions  are  too  complex  to  permit 
simple  closed-form  representations  as  shown  in  Eqs.  (3.4)  and 
(3.5);  rather,  CDFs  and  CCDFs  must  be  determined  through  the 
use  of  various  numerical  procedures. 

Other  summary  measures  for  the  distribution  of  x  (i.e.,  for  the 
probability  space  (X,  X,  pE))  include  the  expected  value  Ex(x)  for  x, 
the  variance  Vx(x)  for  x,  and  the  q  quantile  Qxq(x)  for  x,  where 

Ex(x)  =  jjcdx(x)dx,  (3.6) 

Vx(x)  =  Jjx-Ex(x)fdx(x )  dx,  (3.7) 

and  Qxq(x)  corresponds  to  the  value  of  x  for  which 
q  =  px{Ux)  =  jjjx)dx(x)  dx.  (3.8) 


Fig.  3.1.  Example  CDF  and  CCDF  for  variable  x  with  (i)  a  loguniform  distribution 
on  [2,10]  and  (ii)  p*(x  <x)  and  px(x < x )  used  as  mnemonics  for  the  probabilities 
Px(Ux)  and  px(Wj)  defined  in  conjunction  with  Eq.  (3.2). 
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Conceptually,  the  q  quantile  Qxq(x)  corresponds  to  the  value  of 
x  obtained  by  (i)  starting  at  q  on  the  ordinate  of  the  CDF  for  x, 
(ii)  drawing  a  horizontal  line  to  the  CDF,  and  (iii)  then  drawing  a 
vertical  line  down  to  the  abscissa.  The  value  for  x  at  the  point 
where  the  indicated  vertical  line  intersects  the  abscissa  corre¬ 
sponds  to  the  q  quantile  Qxq(x )  for  x  (Fig.  3.1). 

In  most  analyses,  the  result  of  interest  is  a  function 

y  =/(x)  (3.9) 

of  uncertain  analysis  inputs.  If  x  is  uncertain  as  quantified  by  a 
probability  space  {X,  X,  px ),  then  y  is  also  uncertain,  with  this 
uncertainty  quantified  by  a  probability  space  (y,  Y,  pY )  that 
derives  from  the  function  /(x)  and  the  probability  space  {X,  X, 
Px)  for  x.  In  concept,  it  is  possible  to  derive  the  probability  space 
( y ,  Y,  pY).  In  practice,  (y,  Y,  pY )  is  usually  approximated  with 
sampling-based  procedures  [55,56], 

If  y  corresponds  to  a  scalar  y  or  y  is  a  component  of  the  vector 
y,  then  the  uncertainty  in  y  that  derives  from  the  uncertainty  in  x 
is  usually  represented  by  a  CDF  or  a  CCDF  that  summarizes  the 
corresponding  probability  space  (y,  Y,  pY)  for  y.  Specifically,  the 
CDF  and  CCDF  for  y  are  defined  by  plots  of  the  points 

\y,PY(Uy)]  and  \y,PY<Ucy)],  (3.10) 

respectively,  for y  ey,  where 

Uy  =  {y-yey  and  y^y), 

pY(Uy)  =  probability  of  Uy  (i.e.,  of  a  value  y  <  y) 

=  jj>y\f(x)\dx(x)dX  3  5y\f(Xi)ynSX  =  pY(Uy), 


pY(Ucy)  =  probability  of  Ucy  (i.e.,  of  a  value  y  >y) 

=  jlyf(x)]dx(x)  dX  3  J2  Sy[f(Xt)]/nfflf = PvQ^yl 

5  and  Sy  are  defined  analogously  to  Sx  and  6X  in  conjunction  with 

Eq.  (3.2),  and  x„  i  =  1,2 . nSX,  is  a  sample  from  X  generated  in  a 

manner  consistent  with  the  probability  space  (X,  X,  px ).  The 
sampling-based  (i.e.,  Monte  Carlo)  approximations  to  pY(Uy)  and 
pY(Uy)  are  introduced  because,  in  general,  the  defining  integrals 
for  pY(Uy)  and  pY(pty)  will  be  high-dimensional  and  thus  too 
complex  for  a  closed-form  or  quadrature-based  evaluation. 

As  an  example,  approximations  to  the  CDF  and  CCDF  for 

y=/(x)  =  x?  + 2x1x2+x|  (3.11) 

generated  with  a  random  sample 

X;  =  [xn,xa],  f  =  1,2 . nSX  —  1 00,  (3.12) 

from  uniform  distributions  on  [0,2]  for  x,  and  x2  are  shown  in 
Fig.  3.2. 

Similarly  to  the  summary  measures  for  x  in  Eqs.  (3.6)— (3.8), 
additional  summary  measures  for  the  distribution  ofy  =/(x)  (i.e., 
for  the  probability  space  (y,  Y,  pY))  include  the  expected  value 
Ey(y)  fory,  the  variance  VY (y)  fory,  and  the  q  quantile  Q.Yq(y)  for  y, 
where 

Ey(y)  =  Ex{f(x)]  =  Jj  (x)dx(x)  dX,  (3.13) 

VY(y)  =  Vx[f(x)]  =  J^{f(x)—Ex]f (x)]}2dx(x)  dX,  (3.14) 


y  =  f(x) 


Fig.  3.2.  Example  CDF  and  CCDF  for  y=/(x)=xj+2x,  x  2+x)  generated  with  (i)  a 

random  sample  Xj  =  [xn,Xa],  i  =  1 ,2 . 100,  from  uniform  distributions  on  [0,2]  for 

x,  and  x2  and  (ii)  px[f(x)  <y]  and  px[y  < /(x)]  used  as  mnemonics  for  the  estimated 
probabilities  py(Wy)  and  PyUJ[: )  defined  in  conjunction  with  Eq.  (3.10)  to  empha¬ 
size  the  dependence  of  py{Uy)  and  py(Wy)  on  the  probability  space  {X,  X,  px). 

and  Qyq(y)  =  Qxq(f(xn  corresponds  to  the  value  of  y  for  which 

q  =  pY(Uy)  =  ^6y[f(x)]dx(x)  dX.  (3.15) 

In  practice,  Qyq(y)  is  usually  approximated  by  the  value  y  such 
that 

q  =  pY(Uy)  3  jjr  <5y[f(Xj)]/nSX,  (3.16) 

where  Xj,  i=l,2,  ....  nSX,  is  a  sample  from  X  generated  in  a 
manner  consistent  with  the  probability  space  {X,  X,  px)  as 
illustrated  in  Fig.  3.2.  Further,  Ey(y)  and  Vy(y)  can  be  approxi¬ 
mated  in  a  similar  manner  with  the  indicated  sample. 

3.2.  Basic  entities  underlying  an  analysis 

The  posing  and  answering  of  Questions  (Q1)-(Q4)  introduced 
in  Section  2  gives  rise  to  an  analysis  predicated  on  three  basic 
entities  [98,99]: 

A  probabilistic  characterization  of  aleatory  uncertainty,  (EN1) 
A  model  that  predicts  system  behavior,  (EN2) 

and 

A  probabilistic  characterization  of  epistemic  uncertainty.  (EN3) 

Formally,  EN1  corresponds  to  a  probability  space  ( A ,  A,  pA)  for 
aleatory  uncertainty  and  provides  the  answers  to  Questions  Q1  and 
Q2:  “What  can  happen?”  and  “How  likely  is  it  to  happen?”;  EN2 
corresponds  to  a  function  /  (e.g.,  the  solution  of  a  system  of 
differential  or  partial  differential  equations)  that  determines  analysis 
outcomes  of  interest  and  provides  the  answer  to  Question  Q3: 
“What  are  the  consequences  if  it  does  happen?”;  and  EN3  corre¬ 
sponds  to  a  probability  space  (£,  E ,pE)  for  epistemic  uncertainty  and 
provides  the  basis  for  answering  Question  Q4:  “How  much  con¬ 
fidence  exists  in  the  answers  to  the  first  three  questions?” 

The  sample  space  A  for  the  probability  space  ( A ,  A,  pA)  for 
aleatory  uncertainty  is  a  set  of  the  form 

A={a:a  =  [ai,a2 . a^]},  (3.17) 

where  each  vector  a  contains  the  defining  properties  (e.g.,  time, 
size,  location, . . . )  for  a  single  random  occurrence  associated  with 
the  system  under  study.  In  practice,  (A,  A,  pA)  is  usually  defined 
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by  specifying  probability  distributions  that  characterize  the 
occurrence  of  the  individual  components  of  a  and  hence  the 
occurrence  of  the  individual  elements  of  A.  Further,  the  value  of 
nA  (i.e.,  the  dimension  of  a)  may  change  for  different  elements  of 
A.  For  example,  the  elements  of  A  might  be  of  the  form 
a  =  [n,t1,p1,t2,p2,...,tn,pn],  (3.18) 

where  n  is  the  number  of  occurrences  of  a  Poisson  process  over  a 
specified  period  of  time,  t,-  is  the  time  of  the  ith  occurrence,  and  p, 
is  a  vector  of  properties  associated  with  the  ith  occurrence.  When 
needed,  the  density  function  associated  with  the  probability  space 
{A,  A,  pA)  is  represented  by  dA{ a). 

Similarly,  the  sample  space  £  for  the  probability  space  {£,  E,  pE) 
for  epistemic  uncertainty  is  a  set  of  the  form 
£={e:e  =  [ei,e2 . en£]},  (3.19) 

where  each  vector  e  contains  possible  values  for  the  nE  epistemi- 
cally  uncertain  variables  under  consideration.  When  needed,  the 
density  function  associated  with  the  probability  space  {£,  E,  pE)  is 
represented  by  d£(e). 

In  practice,  {£,  E,  pE)  is  usually  defined  by  specifying  prob¬ 
ability  distributions  that  characterize  the  epistemic  uncertainty 
associated  with  the  individual  components  of  e.  Specifically,  the 
distributions  for  the  elements  of  e  are  providing  a  quantitative 
characterization  of  degrees  of  belief  based  on  all  available 
information  with  respect  to  where  appropriate  values  of  these 
elements  are  located  for  use  in  the  analysis  under  consideration. 
The  development  of  these  distributions  often  involves  an  exten¬ 
sive  expert  review  process  [100-106].  The  importance  of  expert 
review  and  judgment  in  the  characterization  of  epistemic  uncer¬ 
tainty  is  specifically  recognized  in  the  NAS/NRC  report  on  QMU 
(Finding  1-5,  p.  30,  Ref.  [77]). 

In  many  analyses,  e  has  the  form 
e=[e„,eM],  (3.20) 

where 

e/l  =  [eA1  .eA2.-  ■  -.e/l.nr/l] 

is  a  vector  of  nEA  epistemically  uncertain  quantities  used  in  the 
characterization  of  aleatory  uncertainty  (e.g.,  an  imprecisely 
known  rate  1  that  defines  a  Poisson  process)  and 

=  [®M1  ,*M2,-  ■  -.CM.nEM] 

is  a  vector  of  nEM  epistemically  uncertain  quantities  used  in  the 
modeling  of  one  or  more  physical  processes  (e.g.,  an  imprecisely 
known  thermal  conductivity). 

In  some  situations  involving  margin  analyses,  it  may  be 
appropriate  to  further  decompose  eM  into 
eM  =  [eR,eP],  (3.21) 

where  eR  is  a  vector  of  epistemically  uncertain  quantities  used  in 
the  definition  of  the  requirements  that  underlie  the  margins 
under  consideration  and  eP  is  a  vector  of  epistemically  uncertain 
quantities  that  correspond  to  model  parameters.  The  possible 
presence  of  epistemic  uncertainty  in  the  definition  of  require¬ 
ments  is  specifically  recognized  by  the  NNSA  in  Quote  (NNSA-3). 
Also,  the  NAS/NRC  report  on  QMU  recognizes  the  possibility  of 
epistemic  uncertainty  in  a  requirement  in  a  notional  example 
involving  the  determination  of  a  margin  (pp.  25-26,  Ref.  [77]). 

When  e  has  the  form  e=[e„,  eM]  indicated  in  Eq.  (3.20),  the 
analysis  in  effect  has  two  probability  spaces  for  epistemic  uncer¬ 
tainty:  a  probability  space  (£A,  EA,  p^)  that  characterizes  the 
uncertainty  in  eA,  and  a  probability  space  (£M,  EMI,  pEM)  that 
characterizes  the  uncertainty  in  eM.  In  turn, 

(3.22) 


is  the  sample  space  for  the  probability  space  (£,  E,  pE).  In  practice, 
the  probability  spaces  {EA,  EA,  pM)  and  {EM,  EM,  pEM)  are 
defined  by  assigning  distributions  to  the  components  of  and 
eM,  respectively,  which  in  effect  also  defines  the  probability  space 
{£,  E,  pF_).  Although  the  probability  spaces  (£A,  EA,  pEA)  and  {£M, 
EM,  pEM )  are  incorporated  into  the  probability  space  {£,  E,  pE),  it 
is  often  convenient  to  maintain  their  separate  identities  for 
conceptual  and  notational  purposes.  When  needed,  the  density 
functions  associated  with  {EA,  EA,  Pea)  and  {£M,  EM,  pEM)  are 
represented  by  d^e^)  and  dEM{eM),  respectively.  As  a  reminder,  a 
different  probability  space  {A,  A,  pA )  for  aleatory  uncertainty  with 
corresponding  density  function  dA(a  e^ )  results  for  each  element 
eA  of  EA. 

As  previously  indicated,  the  probability  spaces  {A,  A,  pA)  and 
{£,  E,  pE)  correspond  to  the  Entities  EN1  and  EN3.  In  turn,  Entity 
EN2  corresponds  to  a  function  of  the  form 
y(t|a,eM)  =  [jqftla.eM),  y2(t|a,eM)„ .  .,y„y(t|a,eM)]  =  f(t|a,eM), 

(3.23) 

where  as  A,  £  =  £Ax  EM  as  indicated  in  Eq.  (3.22),  eM  e EM,  t 
corresponds  to  time  with  the  assumption  that  time-dependent 
results  are  under  consideration,  and  the  vertical  line  in  y(t|a,  eM) 
is  used  to  indicate  the  concept  of  “conditional  on”.  Further,  in  a 
QMU  analysis,  one  or  more  elements  of  y(t|a,  eM)  will  either  be 
margins  or  analysis  results  used  in  the  definition  of  margins.  In 
most  real  analyses,  the  number  of  results  under  consideration 
(i.e.,  nY)  is  likely  to  be  very  large.  However,  for  notational 
simplicity,  a  real-valued  result 

y(t|a,eM)=/(t|a,eM)  (3.24) 

is  assumed  to  be  under  consideration. 

The  uncertainty  associated  with  y(t|a,  eM)  is  often  studied  in 
one  of  two  contexts.  In  the  first  context,  a  is  assumed  to  be  fixed, 
and  the  uncertainty  in  y(t|a,  eM)  that  derives  from  the  epistemic 
uncertainty  associated  with  eM  is  analyzed.  In  essence,  this 
context  involves  only  the  Entity  EN2  corresponding  to  the  func¬ 
tion  y(t|  a,  eM)  and  the  Entity  EN3  corresponding  to  the  probability 
space  {EM,  EM,  pEM)  that  characterizes  the  epistemic  uncertainty 
associated  with  eM.  The  Entity  EN1  corresponding  to  the  prob¬ 
ability  space  {A,  A,  pA)  that  characterizes  the  aleatory  uncertainty 
associated  with  a  does  not  enter  into  the  analysis  as  a  result  of 
fixing  a  at  a  specific  value. 

In  the  second  context,  a  is  not  assumed  to  be  fixed,  and  the 
distributions  of  y(t|a,  eM)  that  derive  from  the  aleatory  uncer¬ 
tainty  associated  with  a  characterized  by  probability  spaces 
{A,  A,  pA)  conditional  on  specific  values  for  e=[eA  eM]  are  central 
to  the  analysis.  In  this  context,  all  three  entities  are  present,  with  the 
analysis  involving  distributions  that  derive  from  epistemic  uncer¬ 
tainty  for  (i)  CDFs  and  CCDFs  that  derive  from  aleatory  uncertainty 
or  (ii)  summary  quantities  (e.g.,  expected  values,  quantiles)  for  CDFs 
and  CCDFs  that  derive  from  aleatory  uncertainty.  Specifically,  each 
CDF  and  each  CCDF  indicated  in  the  preceding  sentence  derives 
from  aleatory  uncertainty  conditional  on  a  specific  value  for 
e=[e/i,  eM]:  in  turn,  the  epistemic  uncertainty  associated  with  e 
and  characterized  by  the  probability  space  {£,  E,  pE)  results  in 
distributions  of  these  CDFs  and  CCDFs  and  also  in  distributions  of 
quantities  such  as  means  and  variances  that  summarize  these 
CDFs  and  CCDFs.  The  preceding  distributions  that  derive  from 
epistemic  uncertainty  are  the  focus  of  study  in  this  second 
analysis  context. 

The  two  indicated  analysis  contexts  are  discussed  in  the 
next  four  sections  (Sections  3.3-3.6).  However,  most  large  ana¬ 
lyses  that  involve  both  aleatory  uncertainty  and  epistemic  uncer¬ 
tainty  will  have  various  subanalyses  that  involve  each  of  these 
analysis  contexts.  Specifically,  some  subanalyses  will  be  carried 
out  conditional  on  specific  realizations  of  aleatory  uncertainty 
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(i.e.,  analyses  in  the  sense  of  the  first  context  as  discussed  in 
Sections  3.3  and  3.4)  and  some  subanalyses  will  be  carried  out 
that  address  the  epistemic  uncertainty  associated  with  results 
that  derive  from  aleatory  uncertainty  (i.e.,  analyses  in  the  sense  of 
the  second  context  as  discussed  in  Sections  3.5  and  3.6). 

3.3.  Analysis  in  the  presence  of  only  epistemic  uncertainty 

This  section  presents  a  formal  description  of  the  representa¬ 
tion  of  uncertainty  in  an  analysis  that  involves  only  epistemic 
uncertainty.  The  following  section  (Section  3.4)  then  presents  a 
simple  example  illustrating  the  formal  concepts  presented  in  the 
present  section.  If  desired,  Section  3.4  can  be  read  before  Section 

3.3,  with  Section  3.3  being  referred  to  only  when  a  more  technical 
description  of  the  results  in  Section  3.4  is  desired.  The  importance 
of  the  quantification  of  the  epistemic  uncertainty  in  analysis 
results  that  derives  from  epistemic  uncertainty  in  analysis  inputs 
is  emphasized  in  the  NAS/NRC  report  on  QMU  (Recommendation 
1-2,  p.  22,  Ref.  [77]). 

The  CDF  and  CCDF  introduced  in  the  first  analysis  context  at 
the  end  of  the  preceding  section  and  conditional  on  specific 
values  for  t  and  a  are  defined  as  indicated  in  Eq.  (3.10).  Specifi¬ 
cally,  the  CDF  and  CCDF  for  y(t|a,  Cm)  that  derive  from  the 
different  possible  values  for  eM  are  defined  by  plots  of  the  points 
{y,P£M[Wy(t|a)]}  and  {y,pEM[Z4(t|a)]},  (3.25) 

respectively,  for  y  e  y(t  a),  where 
y(t|a)  =  {y  :y  =  y(t|a,eM)  foreMe£M), 

Uy(t  I  a)  =  {y  :  y  e  y(t|a)  and  y  <y}, 

Pm\Uy(t\a)]  =  probability  of  Uy(t [a)  (i.e.,  of  a  valuey  <y) 

=  I  a,  eM  )]dtM(eM )  dEM 

=  J2  Sy\y(t  a,eMi)]/nSE 
=  PbmP/yCT  I  a)l- 

PEM[Ucy(t |a)]  =  probability  of  Wy(t|a)  (i.e.,  of  a  valuey  >y) 

=  J^^Sy\y(t\3,eM)]dEM(CM)<lEM 

=  f^5y[y(t|a,eMi)]/nSE 

5y  and  6y  are  defined  analogously  to  (5xand  <5X  in  conjunction  with 
Eq.  (3.2),  and  eMi,  i=  1,2,  ....  nSE,  is  a  sample  from  EM  generated 
in  a  manner  consistent  with  the  probability  space  (EM,  EM,  pEM ) 
and  its  associated  density  function  dEM(eM).  The  result  is  a  CDF 
and  CCDF  of  the  form  shown  in  Fig.  3.2  that  summarize  the 
epistemic  uncertainty  iny(t|a,eM)  that  derives  from  the  epistemic 
uncertainty  in  eM  characterized  by  the  probability  space  (EM, 
EM,  Pem). 

The  CDF  and  CCDF  defined  in  Eq.  (3.25)  can  also  be  summar¬ 
ized  with  various  real-valued  quantities,  including  an  expected 
value  Et-M[y(t|a,eM)],  a  variance  14m [y(f  a,eM)],  and  selected  quan¬ 
tiles  Q£M<j[y(t|a.eM)]-  As  described  in  Eqs.  (3.13)— (3.16), 

EEM\y(t\a,eM)]  =  J^y(t\a,eM)dEM(eM)dEM 
=  J2y(t  a'  eMi)/nSE 

=  £EM[y(t|a,eM)],  (3.26) 


V4M[y(t|a,eM)]  =  J^{y(t\a,eM)-EEM\y(t\a,eM)]}2dEM(eM)<iEM 
P  J2  ly(t  |  a, ejvfi)  -£em [y( 1 1  a, eM)] ) 2  / nSE 
=  VWy(t|a,eM)],  (3.27) 

and  Q£Mij[y(t|a,  eM)  corresponds  to  the  value  y  such  that 
q  =  pEM[Uy(t\a)]  =  J^5y\y(t\a,eM)]dEM(eM)AEM 

=  2^yWla.eMi)]/nS£,  (3.28) 

where  eMi,  *=1,2 . nSE,  is  the  sample  indicated  in  conjunction 

with  Eq.  (3.25). 

As  discussed  in  Ref.  [56],  the  sample  eMi,  i=  1,2 . nSE, 

also  provides  the  basis  for  the  implementation  of  a  variety  of 
sensitivity  analysis  procedures.  The  use  of  such  procedures 
is  a  natural  and  important  part  of  any  sampling-based  uncer¬ 
tainty  analysis.  The  importance  and  usefulness  of  appropriate 
sensitivity  analyses  is  emphasized  in  the  NAS/NRC  report  on  QMU 
(pp.  14-15,  Ref.  [77]). 

3.4.  Example  analysis  in  the  presence  of  only  epistemic  uncertainty 

A  simple  example  is  now  presented  to  illustrate  the  concepts 
introduced  in  Section  3.3.  This  example  will  also  be  used  in  Section  4 
of  this  presentation  and  in  Section  3  of  Ref.  [80]  to  illustrate 
potential  QMU  analyses  involving  only  epistemic  uncertainty. 

The  example  is  based  on  a  closed  electrical  circuit  that  is  under 
consideration  for  some  unstated  realization  a  of  aleatory  uncertainty. 
For  example,  a  might  simply  correspond  to  nominal  (i.e.,  unper¬ 
turbed)  conditions  for  the  system  under  study.  Specifically,  the 
behavior  of  this  circuit  is  described  by  a  differential  equation 

Ld2Q/dt2 +R  dQ/dt+  Q/C  =  E0  exp(-/t), 

Q(0)  =  0,  dQ(0) /At  =  0,  (3.29) 

where  (i)  Q(t)  is  the  electrical  charge  (coulombs)  at  time  t  (s),  (ii)  L  is 
the  inductance  (henrys),  (iii)  R  is  the  resistance  (ohms),  (iv)  C  is  the 
capacitance  (farads),  (v)  E0  exp(  -  It)  is  the  electromotive  force  (volts), 
and  (vi)  dQ/dt  is  the  current  (amperes).  For  this  example,  it  is  also 
assumed  that  R,  L  and  C  have  values  such  that  the  inequality 
R2— 4L/C  <  0  (3.30) 

holds. 

The  significance  of  the  preceding  inequality  is  that  it  results  in 
Q(t)  displaying  a  damped,  oscillatory  behavior.  In  particular, 
the  closed-form  solution  to  Eq.  (3.29)  when  the  inequality  in 
Eq.  (3.30)  holds  is 

Q(t)  =  exp[(-R/2I)t]{Cl  cos[(y|R2— 4L/C]/2l)  t] 

+  c2  sin[(y|R2-4I/C|/2L)  t]  J-Cl  exp(-2t)  (3.31) 

with 

Ci  =  -CE0/(CLA2-CRA+ 1) 

2  L  ^  I"  ICEp _ RCE0  ] 

Cl  ~  \J\R2—4L/C\  X  [CU2-CR2+1  21(01^80+  1)J  ' 

As  an  example,  Q£t)  is  illustrated  in  Fig.  3.3  with  1=1  henry, 
R=100  ohms,  C=10  4  farads,  £o=1000  volts  and  2=0.1  s 

For  the  examples  of  this  section,  the  vector  eM  of  epistemi- 
cally  uncertain  analysis  inputs  for  the  model  defined  in 
Eqs.  (3.29)-(3.31)  is 
Cm  =  [Cjwi ,  Cm2  ,  Cm3  ,  Cjw4  »  ^ms]  =  [l,R,C,£o,2], 


(3.32) 
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t :  Time  (s) 

Fig.  3.3.  Solution  Q(r)  shown  in  Eq.  (3.31 )  to  differential  equation  in  Eq.  (3.29)  obtained 
with  L=1  henry,  R=  100  ohms,  C=10  4  farads,  E„=  1000  volts,  and  2=0.1  s 


with  eM1,eM2 . eM5  used  in  place  of  L,R . A  to  represent  the 

elements  of  eM  when  notationally  convenient.  Incorporation  of 
a  and  eM  into  the  notation  for  Q(t)  results  in  the  representation 
Q(t|a,eM),  with  Q(t|a,eM)  corresponding  to  the  generic  represen¬ 
tation  y(t|a,eM)  in  Eq.  (3.24). 

The  appropriate  values  for  L,  R,  C,  E0  and  /.  are  assumed  to  be 
contained  in  the  intervals 

EMt  =  {L  :  Lmn  <L< L^}  =  (L  :  0.8  < L <  1.2  henrys),  (3.33) 

EM2  =  {R  :Rmn<R<  Rmx }  =  {R  :  50  <  R  <  100  ohms},  (3.34) 

£M3  =  {C  :  Cmn  <C<  Cmx)  =  {C  :  0.9  x  10-4  <  C <  1.1  x  10~4  farads}, 

(3.35) 

£M4  =  {Eo  :  Emn  <E0<  Emx]  =  {E0  :  900  <  E0  <  1 1 00  volts},  (3.36) 
and 

EM5  =  {A  :  Amn  <  A<  Am)  =  {A:0.4<A<  0.8s-1},  (3.37) 

respectively. 

A  probabilistic  characterization  of  the  epistemic  uncertainty 
associated  with  L,  R,  A  is  provided  by  a  probability  distribution 
defined  on  each  of  the  preceding  intervals.  Specifically,  four 
subintervals  are  considered  for  each  of  the  intervals  £M»  i=  1,2, 
....  5,  defined  in  Eqs.  (3.33)-(3.37): 


£n  =  [a,b-(b-a)/4]. 

(3.38) 

£a  =  [a+(b-a)/4,b]. 

(3.39) 

£a  =  [ct+(b-a)/8,b-3(b-a)/8]. 

(3.40) 

£i4  =  [a+3(b-a)/8,b-(b-a)/8], 

(3.41) 

where  [a,  b]  corresponds  to  [Lmn,  Lmx\,  [Rmn,  Rm*],  [Cmn,  Cmx],  [Emn, 
Emx]  and  [Amn,  Amx]  for  i=  1, 2, 3, 4  and  5,  respectively  (Fig.  3.4).  For 
example  and  for  a  given  element  eMi  of  eM,  each  of  the  preceding 
intervals  could  have  been  indicated  by  a  different  source  as 
containing  the  correct  value  to  use  for  eMi  in  the  analysis  under 
consideration. 

In  turn,  the  corresponding  density  function  d,(eMl)  for  the  set 
£Mi  is  given  by 

di(eMd=  ^ij(eu)/ 4[max(£s)-min(%)]  (3.42) 

i  =  i 

under  the  assumption  that  the  four  sources  that  provided  the 
intervals  in  Eqs.  (3.38)-(3.41)  for  an  element  eMi  of  eM  are  equally 


012345678 


£/i :  -* - *~ 

£i2:- - - 

£f4:  ** 

Fig.  3.4.  Illustration  of  sets  £a,  £a,  £q  and  £,4  defined  in  Eqs.  (3.38)-(3.41)  with 
the  interval  [a,  b]  normalized  to  the  interval  [0,8]  for  representational  simplicity. 


credible,  where 

*  .  _  /  1  if  eMi  e  £ij 

“  j  0  otherwise. 

The  preceding  specification  for  d,(eMi)  corresponds  to  defining  a 
uniform  distribution  on  each  interval  £y  and  then  weighting  each 
distribution  equally.  The  equal  weighting  derives  from  an 
assumption  of  equal  credibility  for  the  four  sources  of  the 
intervals  in  Eqs.  (3.38)-(3.41).  The  definition  of  the  density 
functions  di(eMi)  in  Eq.  (3.42)  results  in  the  assignment  of  more 
probability  where  the  intervals  supplied  by  the  four  sources 
overlap  and  less  probability  where  the  intervals  do  not  overlap. 
The  density  functions  d,(eMi)  in  essence  define  probability  spaces 
(£Mj,  EMIj,  pEMJ)  for  the  variables  eMi,  i=  1,2 . 5. 

The  set  £M  of  possible  values  for  eM  is  given  by 

£M  =  £M i  x  £M2  x  £M3  x  £M4  x  £M5,  (3.43) 

where  £M\,  £M2 . £Ms  are  defined  in  Eqs.  (3.33)-(3.37).  In 

turn,  £M  has  a  probabilistic  structure  that  derives  from  the 
distributions  characterizing  the  uncertainty  in  eM1,  eM2,  ....  eM5. 
Formally,  this  structure  corresponds  to  a  probability  space  (£M, 
EMI,  pEM )  that,  in  effect,  is  defined  by  the  density  functions 
introduced  in  Eq.  (3.42). 

The  epistemic  uncertainty  associated  with  Q(t|a,  eM)  that 
derives  from  the  probability  space  (£M,  EM,  pEM)  is  now 
considered.  As  indicated  in  Eqs.  (3.25)-(3.28),  this  uncertainty 
can  be  characterized  by  various  quantities  defined  by  integrals 
over  £M.  However,  such  integrals  are  difficult  to  determine  in 
closed  form  (i.e.,  by  use  of  antiderivatives  in  conjunction  with  the 
Fundamental  Theorem  of  Calculus)  because  of  the  high  dimen¬ 
sionality  of  eM  and  the  complexity  of  the  function  being  inte¬ 
grated.  Instead,  sampling-based  methods  are  used  in  most 
analyses  to  determine  these  quantities. 

Consistent  with  this  approach,  the  present  example  uses  a 
Latin  hypercube  sample  (LHS) 

Cmi  =  [em,i>eM2,i . eM5,i].  i=1,2 . nSE,  (3.44) 

of  size  nSE=  200  generated  from  £M  in  consistency  with  the 
probability  space  (£M,  EM,  pEM)  (i.e.,  in  consistency  with  the 
distributions  associated  with  the  density  functions  defined  in 
Eq.  (3.42)).  As  discussed  in  Refs.  [45,55,56],  Latin  hypercube  sam¬ 
pling  is  widely  used  in  sampling-based  uncertainty  and  sensitivity 
analyses  involving  computationally  demanding  models  because  of 
its  efficient  stratification  properties.  In  addition,  this  sampling-based 
approach  also  provides  the  basis  for  the  application  of  a  variety  of 
sensitivity  analysis  procedures  [56], 

The  sample  in  Eq.  (3.44)  results  in  nSE=  200  time-dependent 
results:  Q(t|a,  eMi),  i=l,2,  ...  200  (Fig.  3.5).  The  spread  of  the 
curves  in  Fig.  3.5  provides  a  nonquantitative  indication  of  the 
epistemic  uncertainty  associated  with  Q(t|a,  eM)  that  derives 
from  the  uncertainty  in  eM  as  quantified  by  the  probability  space 
{£M,  EM,  pEM).  Only  50  of  the  200  time-dependent  results  for 
Q(t|a,  eMj)  are  presented  in  Fig.  3.5  as  presentation  of  all  200 
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Q :  Q(0.1|a,  eM) 

Fig.  3.6.  Estimated  CDF  and  CCDF  for  Q(0.1  |a,  e„.)  (i)  obtained  with  the  LHS  of  size 
200  in  Eq.  (3.44)  generated  from  EM  in  consistency  with  the  defining  density 
functions  for  the  probability  space  (EM,  EM,  pfM)  and  (ii)  presented  with 
PemIQL 0.1 1 a,  eM)  <  Q]  and  p£JW[Q<  Q(0.1  |a,  eM)]  used  as  mnemonics  for  estimated 
probabilities  of  the  form  p£M[W y(0.1  |a)]  and  Pem[^( 0-1  |a)]  defined  in  conjunction 
with  Eq.  (3.25). 


belief’  probability  of  0.9  that  Q(0.1  |a,eM)  is  located  between  the  0.05 
and  0.95  quantiles  in  Fig.  3.6.  Further,  the  expected  value  and  quantile 
values  indicated  in  Fig.  3.6  are  obtained  as  described  in  Eqs.  (3.26) 
and  (3.28),  respectively.  In  this  example,  the  expected  value  and 
median  value  are  very  close  together;  this  is  often  not  the  case  in 
analyses  that  involve  substantial  epistemic  uncertainties. 

The  presentation  of  CDFs  and  CCDFs  of  the  form  shown  in 
Fig.  3.6  for  multiple  values  of  t  is  cumbersome.  An  effective 
alternative  is  to  plot  expected  values  and  quantiles  as  functions  of 
time  (Fig.  3.7).  With  this  presentation  format,  expected  values  and 
quantiles  as  indicated  in  Fig.  3.6  are  determined  for  a  sequence  of 
values  for  t  and  then  plotted  above  these  values  to  obtain  the 
expected  value  and  quantile  curves  in  Fig.  3.7.  Specifically,  the 
expected  value  curve  in  Fig.  3.7  is  a  plot  of  the  points 

(t,££M[y(t|a,eM)]),  0  <  t  <  0.20,  (3.45) 

with  Eh:M\y(t  a,eM)]  defined  as  indicated  in  Eq.  (3.26),  and  the 
quantile  curves  in  Fig.  3.7  are  plots  of  points 

(t.Q£M,[y(t|a.eM)]),  0  <  t  <  0.20,  (3.46) 

for  q=  0.05,  0.5  and  0.95  with  Q£MlJ[y(t|a,eM)]  defined  as  indicated 
in  Eq.  (3.28).  In  this  example,  the  expected  value  and  median 
value  (i.e.,  0.5  quantile)  curves  almost  exactly  overlap. 

3.5.  Analysis  in  the  presence  of  aleatory  and  epistemic  uncertainty 

This  section  presents  a  formal  description  of  the  representa¬ 
tion  of  uncertainty  in  an  analysis  that  involves  both  aleatory  and 
epistemic  uncertainty.  The  following  section  (Section  3.6)  then 
presents  a  simple  example  illustrating  the  formal  concepts  pre¬ 
sented  in  the  present  section.  If  desired,  Section  3.6  can  be  read 
before  Section  3.5,  with  Section  3.5  being  referred  to  only  when  a 
more  technical  description  of  the  results  in  Section  3.6  is  desired. 

The  analyses  described  in  Sections  3.5  and  3.6  involve  what 
the  NAS/NRC  report  on  QMU  refers  to  as  the  “probability  of 
frequency  approach"  and  recommends  for  use  in  QMU  analyses 
(Recommendation  1-7,  p.  33,  and  App.  A,  Ref.  [77]).  Specifically, 
the  descriptor  “probability  of  frequency  approach”  designates  an 
analysis  in  which  a  careful  distinction  and  separation  is  main¬ 
tained  between  the  effects  and  implications  of  aleatory  uncer¬ 
tainty  and  the  effects  and  implications  of  epistemic  uncertainty. 

The  CDF  and  CCDF  introduced  in  the  second  analysis  context 
described  in  Section  3.2  and  conditional  on  specific  values  for 
t  and  e=[eA  eM]  are  also  defined  as  indicated  in  Eq.  (3.10). 


curves  results  in  an  almost  solid  band  of  overlapping  curves  that 
obscures  the  shape  of  the  individual  curves. 

A  quantitative  summary  of  the  epistemic  uncertainty  in  Q(t|a, 
eM)  that  derives  from  the  epistemic  uncertainty  in  eM  is  provided 
by  the  CDFs  and  CCDFs  for  Q(t|a,  eM)  at  selected  points  in  time.  As 
an  example,  approximations  to  the  CDF  and  CCDF  at  t=0.1  s 
obtained  with  use  of  the  sample  in  Eq.  (3.44)  are  shown  in  Fig.  3.6. 
Specifically,  the  CDF  and  CCDF  in  Fig.  3.6  are  constructed  from  the 
values  for  Q(0.1  |a,  eMi)  associated  with  the  vertical  line  in  Fig.  3.5 
as  described  in  conjunction  with  Eq.  (3.25).  With  respect  to 
notation,  the  expressions  p£M[Q{ 0.1  |a,  eM)  <  Q]  and  ptM[Q<  Q 
(0.1 1 a,  eM)]  on  the  ordinate  of  Fig.  3.6  correspond  to  the  defining 
probabilities  for  the  CDF  and  CCDF,  respectively,  for  Q(0.1  |a,  eM). 
Specifically,  p£M[Q( 0.1  |a,  eM)  <  Q]  is  the  estimated  probability  that 
Q(0.1 1 a,  eM)  is  less  than  or  equal  to  a  value  Qon  the  abscissa,  and 
Pem[Q<  0(0.1  |a,  eM)[  is  the  estimated  probability  that  Q{0.1  |a,  eM) 
is  greater  than  a  value  Q  on  the  abscissa.  The  indicated  probabil¬ 
ities  are  characterizing  epistemic  uncertainty  and  thus  are  indi¬ 
cating  degrees  of  belief  with  respect  to  where  the  correct  value  for 
0(0.1 1 a,  eM)  is  located.  Thus,  for  example,  there  is  a  “degree  of 
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Specifically,  the  CDF  and  CCDF  for  y(t|a,  eM)  that  derive  from  the 
different  possible  values  for  a  are  defined  by  the  plots  of  the  points 
[y,pA[Uy(t\e)\eA]}  and  {yj^ftfelle,,]},  (3.47) 

respectively,  for  y  e  y(t|e),  where 
W|e)  =  {y  :y=y(t|a,eM)  for  aeA), 

Wy(t|e)  =  {y  :y  ey(t|e)  and  y<y}, 

P/i | e) | e^i]  =  probability  of  Uy(t\e)  (i.e.,  of  a  valued  <y) 

=  J &Jy(t  |  a,eM)]d/,(a  |  eA)  dA 

=  8y\y(t  a,,eM)]/nS/\ 

=pA[Wy(t|e)|eA], 

pA[W£(t|e)|eA]  =  probability  of  Uy{t\e)  (i.e.,  of  a  value  y  >y) 

=  ^<5y[y(f  a,ejw)]dA(a|eA)dA 

=  X!3y[y(t|aj,eM)]/nS7\ 

=  pA[W5(t|e)|eA], 

8  and  8y  are  defined  analogously  to  <5x  and  5X  in  conjunction  with 

Eq.  (3.2)  and  a,,  j=  1,2 . nSA,  is  a  sample  from  A  generated  in  a 

manner  consistent  with  the  probability  space  {A,  A,  pA)  and  its 
associated  density  function  dA(a|eA).  The  result  is  a  CDF  and  CCDF  of 
the  form  shown  in  Fig.  3.2  that  summarize  the  aleatory  uncertainty  in 
y(t|a,  eM)  that  derives  from  the  aleatory  uncertainty  in  a  characterized 
by  the  probability  space  (A,  A,  pA). 

In  general,  the  set  A  could  be,  and  often  is,  a  function  of  elements 
of  eA.  In  this  case,  the  sets  A  and  A  would  appropriately  be  repre¬ 
sented  by  yf(ea)  and  A(eA).  Then,  the  representation  for  the  prob¬ 
ability  space  {A,  A,  pA)  conditional  on  an  element  e=[eA,  e^]  of 
e  would  be  [_A(e/0,  A(eA),  pA{U\eA)\.  To  reduce  notational  clutter,  this 
fully  general  representation  for  (A  A,  pA)  is  not  used.  Instead,  the 
possible  dependence  of  (A,  A,  pA)  on  the  elements  of  eA  is  indicated 
through  the  use  of  the  notations  pA{U  eA)  and  dA(a|eA). 

Similarly  to  the  CDF  and  CCDF  defined  in  Eq.  (3.10),  the  CDF 
and  CCDF  defined  in  Eq.  (3.47)  can  also  be  summarized  with 
various  real-valued  quantities,  including  an  expected  value 
EA[y(t|a,  eM)|eA],  a  variance  VA[y(t  a,  eM)|eA]  and  selected  quan¬ 
tiles  QA„[y(t|a,  eM)|eA],  The  definitions  of  EA[y{t  a,  eM)|e„], 
Va[y(t|a,  eM)|eA]  and  Qa,[y(t|a,  eM)  eA]  are  effectively  the  same 
as  the  definitions  for  £EM[y(t|a,  eM)],  V™[y(t  a,  eM)]  and 
QfiM,[y(t|a,  eM)l  in  Eqs.  (3.26)-(3.28)  with  the  only  difference  being 
that  integrations  are  performed  with  respect  to  the  probability  space 
(A,  A,  pA)  and  its  associated  density  function  dA(a  eA)  rather  than 
with  respect  to  the  probability  space  {EM,  EM,  pEM)  and  its 
associated  density  function  dEM(eM).  Specifically, 

EA\y(t  |  a,  eM)  |  eA]  =  Jy(t  \  a,eM)dA(a  |  eA)  dA 

=  ^y(t|aj,eM)/nSA 
j  =  i 

=  £A[y(t|a,eM)|eA],  (3.48) 

VA[y(t|a,eM)|eA]  =  J  {y(t|a,eM)-£A[y(t|a,eM)]}2dA(a|eA)dA 
s  I>(t|a,-,  eM)-EA[y(t|a,eM)]}2/nSA 
=  VA\y(t\a,  eM)|eA],  (3.49) 


and  Qa,[y(t|a,  eM)|eA]  and  its  approximation  QA?[y(t|a,  eM)|eA] 
correspond  to  the  value  y  such  that 

q  =  Pa  [Wy(t  |  e)  |  eA]  =  J  8 y\y(t\  a,eM)]dA(a  |  eA)  dA 

=  £5y[y(t|aj,eM)]/nSA,  (3.50) 

where  a,-,  j=l,2 . nSA,  is  the  sample  indicated  in  conjunction  with 

Eq.  (3.47). 

Distributions  of  CDFs  and  CCDFs  result  from  the  different 
possible  values  for  e=[eA,  eM],  As  indicated  in  conjunction  with 
Eq.  (3.47),  each  value  for  e  results  in  a  different  CDF  and 
associated  CCDF  that  summarize  the  effects  of  aleatory  uncer¬ 
tainty.  In  turn,  these  CDFs,  CCDFs  and  their  associated  summary 
measures  have  distributions  that  characterize  epistemic  uncer¬ 
tainty  and  derive  from  the  epistemic  uncertainty  in  e  character¬ 
ized  by  the  probability  space  ( £ ,  E,  pE). 

In  general,  the  probability  space  {£,  E,  pE)  will  result  in 
infinitely  many  CDFs  and  CCDFs  of  the  form  defined  in  conjunc¬ 
tion  with  Eq.  (3.47).  Thus,  some  way  of  summarizing  these 
CDFs  and  CCDFs  is  necessary.  As  illustrated  in  Section  3.6,  this 
summary  is  provided  by  expected  value  curves  and  quantile 
curves  (e.g.,  q=0.05,  0.5,  0.95)  defined  by 
(y,EE{pA[Uy(t | e) | eA]})  and  (y,QEq{pA|7fy(t|e)|eA]})  (3.51) 

and 

(y,EE{pA[U$(t | e) | eA]})  and  (y,QE,{pA^(t|e)|eA]})  (3.52) 

for  distributions  of  CDFs  and  CCDFs,  respectively,  where  (i) 
EE{pA\Uy(t  |  e)  |  eA]}  =  jpA[Uy{t  |  e)  |  eA]dE(e)  d £ 

=  jt.{  jA6y^(t  a.eM)]d/i(a  eA)dAjd£(e)d£ 


=  ££{p„[Wy(t|e)|eA]} 


and 

££{Pa|W5(t[e)|eA]}  =  J  pA[w^(t|e)|eA]d£(e)d£ 

=  JJy  f/yttV  I  a,eM)]dA(a  |  eA)  dA  j  d£(e)  d£ 


=  ££{Pa[Wy(t|e)|eA]}, 


(ii)  Q£<j{Pa[Wy(t|e)|eA]}  and  the  associated  approximation 

Q_Hq{pA[Uy  (t|e)|eA]}  correspond  to  the  probability  p  such  that 

q  =  J£8p{pA[Uy{t\e)\eA]}  dB(e)d £ 

=  L-p{  Ip*  ^ a,eM)]d/l(a  I  e/')dA|dt(e)  d£ 

nSE  (  nSA  1  / 

=  |  ^dyty(t|au.eMi)]/nSA  j  j  nSE, 

(iii)  Qfiq  (Pyi  [Z^5(t  |  e)  |  e>i]}  and  the  associated  approximation 

QfijfPaP^  (t|e)jeA])  correspond  to  the  probability  p  such 
that 

q  =  j^Sp{pA[Ucy(te)\ eA]}d£(e)  d£ 

=  L~p{  I  a'eM,]d/'(a  \e^dA)  dt(e)  d£ 
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(iv)  e,-  =  [e4i,eMi],  i=  1,2 . nSE,  is  a  sample  from  £  generated  in 

a  manner  consistent  with  the  probability  space  {£,  E,  p£)  and 

its  associated  density  function,  and  (v)  a,j,  j=  1,2 . nSA,  is  a 

sample  from  .4  generated  in  a  manner  consistent  with  the 
probability  space  [A,  A,  pA)  and  its  associated  density 
function  d/1(a|e/i1)  for  each  e,.  Although  not  incorporated 
into  the  notation  in  use,  the  sample  size  nSA  could  change 
for  each  e;. 


An  alternative  summary  is  provided  by  reducing  each  CDF  to 
its  corresponding  expected  value  EA[y(t  |a,  eM)|eA]  as  indicated  in 
Eq.  (3.48)  and  then  presenting  the  CDF  and  CCDF  for  £U[j/(t|a, 
e/vi)  |  eA]  that  result  from  the  epistemic  uncertainty  associated  with 
e=[e>i,  eM],  Similarly  to  the  results  presented  in  conjunction  with 
Eq.  (3.10),  the  CDF  and  CCDF  for  EA\y(t\a,eM)  e/i]  are  defined  by 
plots  of  the  points 

{y,pE[Uym  and  {y,Pt[^(f)]),  (3.53) 

respectively,  for  ye^ft),  where 

Xf)  =  ly :  y  =  Fa  ty(t  I  a, eM)  |  e/i]  for  e  =  [ea,  eM]  e  £}, 

Wy(t)  =  {y  :p  ey(t)  and  y  <y), 


p£[ZYy(t)]  =  probability  of  Uy(t)  (i.e.,  of  a  value  y  <y) 
=  j^{EA\y(t\a,eM)\eA]}dE(e)dE 


=  X  iy j  Xj  y<f  I  aij’eMi)/nSA 

=  PE[^y(t)], 


nSE 


Pe[UcJ t)]  =  probability  of  UcJt)  (i.e.,  of  a  value  y  >y) 

=  fgSy{EAW  a,eM)|eA])dt-(e)dE 

nSE  _  (  nSA  1  / 

=  X<M  X^l3!/-  eMi)/nSA  l  /  nSE 

=  PE\UCy(t)l 

and  the  samples  e;= [e,^,  eMi],  i=  1,2 . nSE,  and  a,j,  j=l,2 . 

nSA,  are  defined  the  same  as  indicated  in  conjunction  with 
Eqs.  (3.51)  and  (3.52). 

If  desired,  the  reduction  indicated  in  the  preceding  paragraph 
can  be  carried  further  by  reducing  the  expected  value  £ULy(t|  a, 
eM)|e,4]  over  aleatory  uncertainty  defined  in  Eq.  (3.48)  to  an 
expected  value  E£{E/i[.y(t  |a,  eM)|eA]}  over  aleatory  and  epistemic 
uncertainty  defined  by 
E£{EA[y(t|a,eM)M 

=  J^EA\y(t  |  a,eM)  |  eA]dE(e)  d£ 

=  [jf  y(t | a,eM)dA(a |  e„)  dAj d£(e) d£ 

e  X  ^y(t|ay,eMi)/nSA|y/ nSE 

=  ££{E„[y(t|a,eM)|eA]},  (3.54) 

where  the  samples  eMi],  i=l,2,  ....  nSE,  and  a,,,  j=  1 ,2 . 

nSA,  are  again  defined  the  same  as  in  conjunction  with  Eqs.  (3.51 ) 
and  (3.52).  The  expected  value  ££{£/i[y(t|a,eM)  e/i])  is  the  result  of 
reducing  all  the  information  associated  with  the  probability  space 
(£,  E,  pE)  for  epistemic  uncertainty,  the  probability  space  {A,  A,  p„ ) 


for  aleatory  uncertainty  and  the  function  y(t|a,  eM )  to  a  single 
number. 


3.6.  Example  analysis  in  the  presence  of  aleatory  and  epistemic 
uncertainty 

A  simple,  randomly  perturbed  system  is  now  presented  to 
illustrate  the  concepts  introduced  in  Section  3.5.  This  example 
will  also  be  used  in  Section  5  of  this  presentation  and  in  Section  4 
of  Ref.  [80]  to  illustrate  potential  QMU  analyses  involving  aleatory 
and  epistemic  uncertainty.  Further,  the  results  from  real  analyses 
presented  in  Ref.  [79]  are  of  the  form  described  in  the  present 
section. 

The  system  is  assumed  to  receive  random  perturbations  in 
time  whose  occurrence  is  characterized  as  a  stationary  Poisson 
process  with  a  rate  A  (s-1).  The  amplitudes  (i.e.,  magnitudes)  for 
the  individual  perturbations  are  assumed  to  vary  randomly  and  to 
undergo  exponential  decay  in  a  manner  characterized  by  a  rate 
constant  r  (s-1).  In  concept,  any  of  a  large  variety  of  systems  could 
be  under  consideration,  with  the  result  that  the  perturbation 
might  involve  a  mechanical  force,  an  electrical  impulse,  a  radia¬ 
tion  impulse,  a  heat  impulse,  the  injection  of  a  material,  or  some 
additional  possibility.  For  notational  convenience,  the  initial 
perturbations  will  be  represented  by  A0  and  assumed  to  have 
units  of  force  (kg  m/s2).  As  a  result  of  the  indicated  exponential 

A(t)=A0exp[-r(t-t0)]  (3.55) 

is  the  amplitude  at  time  t  of  a  perturbation  of  size  A0  that  occurs 
at  time  t0.  Further,  the  amplitudes  of  the  individual  perturbations 
are  assumed  to  be  characterized  by  a  triangular  distribution 
defined  on  an  interval  [a,  b]  with  a  mode  of  m. 

This  example  involves  both  aleatory  and  epistemic  uncer¬ 
tainty.  For  a  given  time  interval  (e.g.,  [0,  10  s]),  the  different 
possible  realizations  of  aleatory  uncertainty  correspond  to  vectors 
of  the  form 

a=[n,t1,Aoi,t2,A02,. .  .,t„,A0n],  (3.56) 

where  (i)  n  is  the  number  of  perturbations  that  occur  in  the  time 
interval,  (ii)  ti  <  t2<  ...  <tn  are  the  times  at  which  the  individual 

perturbations  occur,  and  (iii)  A0i,  A02 . A0n  are  the  initial 

amplitudes  for  the  individual  perturbations.  In  turn, 

A=  {a  :  a  =  [n,ti,A0i,t2,A02 . tn,A0n]}  (3.57) 

is  the  sample  space  for  aleatory  uncertainty,  and  the  probabilistic 
structure  required  to  formally  complete  the  definition  of  the 
corresponding  probability  space  (A  A,  pA)  derives  from  A  and 
the  probability  distribution  for  A0. 

For  a  given  element  a  of  A  and  a  given  value  for  r,  the  resultant 
amplitude  A(t|a)  at  time  t  is  given  by 

!0  if  t  <  ti 

±A0kexP[-r(t-tk)]  if  t>tl,  <3-58) 

where  n  =  max{k  :  tk  <  t). 

For  this  example,  A,  a,  m,  b  and  r  are  assumed  to  be  uncertain 
in  an  epistemic  sense.  As  a  result, 

e=  [eA,eM]  =  [ei,e2,e3,e4,e5]  =  [A.a.m.b.r]  (3.59) 

is  the  vector  of  epistemically  uncertain  variables  under  considera¬ 
tion,  with  eA=[A,  a,  m,  b]  and  eM=[r],  Specifically,  A,  a,  m  and  b 
are  involved  in  the  definition  of  probability  distributions  that 
characterize  aleatory  uncertainty,  and  r  relates  to  the  physical 
processes  involved  in  the  decay  of  an  initial  perturbation  A0. 
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The  appropriate  values  for  X,  a,  m,  b  and  r  are  assumed  to  be 
contained  in  the  intervals 

£A\  =  [A  :  Xmn  <X<  Amx]  =  [X  :  0.5  <  l  <  1.5s-1}  (3.60) 


£A2  =  [a  :  amn  < a  <  cimx}  =  {a  :  1.0<a<2.0kgm/s2}  (3.61) 

£A3  =  {m  :mmn<m<  nW  =  {m  :  2.0  <  m  < 4.0  kg  m/s2}  (3.62) 

£At  =  {fa  :  bm„  <,  b  <  brm)  =  {b  :4.0<,b<,  5.0  kg  m/s2}  (3.63) 

£ =  {r  :  rmn  <  r  <  rmx}  =  [r  :  0.2  <  r  <  1 .2 s”1 },  (3.64) 

respectively. 

The  resultant  sample  space  for  the  vector  e  of  epistemically 
uncertain  variables  is 

£  =  £A3  x  £A2  x  £A3  x£Atx  £Mj  (3.65) 


A  :  4(1 0|a,  0.7) 


Fig.  3.8.  Estimated  CDF  and  CCDF  for  amplitude  A(10|a,  0.7)  with  eA=[1.0, 1.5, 3.0, 
4.5]  and  eM=[0.7]  (i)  determined  with  a  sample  of  size  nS4=10,000  from  the  set  A 
of  possible  values  for  a  conditional  on  e-[e„,  eM]-[1.0,  1.5,  3.0,  4.5,  0.7]  and 
(ii)  presented  with  pA[4(10|a,  0.7)<4|eA]  and  pA[A  <7V(10|a,  0.7)|eA]  used  as 
mnemonics  for  estimated  probabilities  of  the  form  pA[Wj,(10|e)|eA]  and 
pA[Wj(10|e)|eA]  defined  in  conjunction  with  Eq.  (3.47). 


with  £A\,  £A2 . £M\  defined  in  Eqs.  (3.60)-(3.64).  Further, 

associated  probability  spaces  {£Ai,  EAjp  PeaJ),  i=  1 ,  2,  3,  4,  and 
{£M\,  EMi,  Pem. i)  for  the  individual  elements  of  e  (i.e.,  X,  a,  m,  b 
and  r)  and  also  the  probability  space  (£,  E,  pE)  for  e  are  defined 
in  the  same  manner  as  for  the  elements  of  e=[L,  R,  C,  E0,  X]  in 
Eqs.  (3.38)-(3.42). 

For  a  given  value  for  e=[2,  a,  m,  b,  r],  a  distribution  for  A(t  a, 
eM)=A(t | a,  r)  over  the  possible  values  for  a  results  for  each  time  t 
as  indicated  in  conjunction  with  Eq.  (3.47).  As  an  example,  the 
CDF  and  CCDF  for  A(t|a,  r)  at  t=10s  conditional  on  e=[1.0,  1.5, 
3.0,  4.5,  0.7]  (i.e.,  for  A(10|a,  0.7))  is  illustrated  in  Fig.  3.8.  The 
results  in  Fig.  3.8  were  generated  with  a  random  sample 

a,,  j=  1,2 . nSA,  (3.66) 

of  size  nSA=  10,000  from  A  obtained  in  consistency  with  the 
distributions  for  perturbation  time  t  and  perturbation  magnitude 
A0  that  derive  from  A=1.0s  \  a=  1.5  kg  m/s2,  m=3.0kgm/s2, 
and  b= 4.5  kg  m/s2  (i.e.,  from  e„=[1.0,  1.5,  3.0,  4.5]).  In  addition, 
estimates  for  the  expected  value  £A[A(10|a,  0.7)|e/i]  and  the 
q=0.9  quantile  Q^.o.glA/lO  a,  0.7)  eA]  for  A(10|a,  0.7)  obtained 
with  the  preceding  sample  as  indicated  in  Eqs.  (3.48)  and  (3.50) 
are  also  shown  in  Fig.  3.8. 

A  subset  of  the  results  used  in  the  generation  of  Fig.  3.8  is 
shown  in  Fig.  3.9.  Each  of  the  curves  in  Fig.  3.9  is  a  plot  of  A(t|aj, 
0.7)  for  0<t<20s  and  a  specific  element  a,  of  the  sample 
indicated  in  Eq.  (3.66).  Specifically,  a  plot  of  A(t|a-i,  0.7)  is  shown 

in  Fig.  3.9a,  and  plots  of  A(t|aj,  0.7)  for  j— 1,2 . 5  are  shown  in 

Fig.  3.9b.  The  CDF  and  CCDF  in  Fig.  3.8  summarize  the  aleatory 
uncertainty  (i.e.,  intrinsic  variability)  in  the  values  for  A(10|a,  0.7) 
associated  with  the  vertical  line  originating  at  t=10  s  in  Fig.  3.9 
for  all  elements  of  the  sample  in  Eq.  (3.66). 

The  CCDF  in  Fig.  3.8  summarizes  the  aleatory  uncertainty  in 
A(t|a,  0.7)  at  t=  10  s  as  indicated  by  the  vertical  line  in  Fig.  3.9. 
Corresponding  summaries  are  possible  for  each  value  of  t  in  the 
interval  under  consideration.  However,  presentation  of  such 
summaries  for  a  large  number  of  values  for  t  is  cumbersome. 
A  more  compact  summary  is  to  present  the  expected  value  for 
A(t|a,  0.7)  and  selected  quantiles  for  A(t|a,  0.7)  (e.g.,  0.05,  0.25, 
0.5,  0.75,  0.95)  as  functions  of  time  (Fig.  3.10).  This  format 
presents  the  primary  uncertainty  information  for  A(t|a,  0.7)  as  a 
function  of  time  in  a  single  plot.  The  expected  values  and 
quantiles  in  Fig.  3.10  are  obtained  from  the  sample  in  Eq.  (3.66) 
as  described  in  Eqs.  (3.51)  and  (3.52)  and  illustrated  in  Fig.  3.8. 

b 


Fig.  3.9.  Illustration  of  time-dependent  amplitudes  A(t|aj,  0.7)  used  in  generation  of  CDF  and  CCDF  in  Fig.  3.8  for  aleatory  uncertainty  in  amplitude  at  t=  10  s  conditional  on 
e=[1.0, 1.5,  3.0.  4.5,  0.7]:  (a)  A(t|%,  0.7)  for  j=l,  and  (b)  A(t|%,  0.7)  for  j=l,2,  ....  5. 


J.C. 
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If  e=[x,  a,  m,  fa,  r]  was  precisely  known,  then  results  of  the 
form  shown  in  Figs.  3.8  and  3.10  would  be  the  unique  outcomes 
of  the  analysis.  However,  e  is  not  precisely  known  and  has  many 
possible  values.  As  a  result,  there  are  many  possible  values  for  the 
results  in  Figs.  3.8  and  3.10.  For  example,  there  are  many  possible 
values  for  the  CDF  and  CCDF  in  Fig.  3.8  (Fig.  3.11),  with  each 
possible  CDF  and  CCDF  deriving  from  a  different  element  e=[l,  a, 
m,  b,  r]  of  the  set  £  defined  in  Eq.  (3.65). 

Specifically,  the  results  in  Fig.  3.11  were  generated  with  an  LHS 

e,-  =  [eAj,eMi]  =  [faauitiubi.n],  i=  1.2 . nSE,  (3.67) 

of  size  nSE =200  from  the  set  £  in  consistency  with  the  distribu¬ 
tions  that  define  the  probability  space  {£,  E,  pE)  for  epistemic 
uncertainty.  In  turn,  a  different  CDF  and  associated  CCDF  results 
for  each  sample  element  e„  Further,  the  individual  CDFs  and 
CCDFs  were  estimated  with  random  samples 

aj  =  [nj,tij,A0]j,t2j,A02j . tnjj,A0rljj],  j=  1,2,. .  „nSA,  (3.68) 

of  size  nSA=  10,000  from  A  generated  in  consistency  with  eA—  a„ 
m,-,  fa,]  and  the  corresponding  probability  space  {A,  A,  pA)  for  aleatory 
uncertainty  and  its  associated  density  function  dA(a\eAi).  Although  not 
explicitly  incorporated  into  the  notation  in  use,  the  set  A  changes  for 


t :  Time  (s) 

Fig.  3.10.  Estimated  expected  value  and  quantile  curves  for  aleatory  uncertainty  in 
amplitude  A(t|a,  0.7)  as  a  function  of  time  conditional  on  e=[1.0, 1.5,  3.0, 4.5,  0.7], 


a 


each  eAi  as  a  result  of  the  effect  of  the  interval  [a,-,  fa,-]  on  the  set  of 
possible  values  for  the  size  of  the  perturbation  associated  with  each 
occurrence  of  the  Poisson  process  under  consideration. 

When  small  exceedance  probabilities  arising  from  aleatory 
uncertainty  are  the  analysis  outcomes  of  interest,  CCDFs  are 
usually  plotted  with  log-transformed  values  on  the  ordinate 
(Fig.  3.12a).  Use  of  log-transformed  values  allows  small  excee¬ 
dance  probabilities  to  be  displayed;  in  contrast,  small  exceedance 
probabilities  are  difficult,  and  sometimes  impossible,  to  deter¬ 
mine  from  plotted  results  when  a  linear  scale  is  used  on  the 
ordinate  (e.g.,  compare  the  CCDFs  in  Figs.  3.11b  and  3.12a).  In 
many  QMU  analyses,  it  is  likely  that  small  exceedance  probabil¬ 
ities  will  be  the  analysis  outcomes  of  greatest  interest. 

Distributions  of  CDFs  and  CCDFs  can  be  summarized  with 
expected  value  and  quantile  curves  as  indicated  in  conjunction 
with  Eqs.  (3.51)  and  (3.52).  As  an  example,  a  summary  of  this 
form  is  presented  in  Fig.  3.12b  for  CCDFs  with  log-transformed 
exceedance  probabilities. 

As  indicated  in  conjunction  with  Eq.  (3.53),  distributions  of 
CDFs  and  CCDFs  can  also  be  summarized  by  reducing  each  CDF 
and  corresponding  CCDF  to  an  expected  value  and  then  present¬ 
ing  the  CDF  and  CCDF  for  the  resultant  expected  values  (Fig.  3.13). 
Specifically,  Fig.  3.13  shows  the  CDF  and  CCDF  for  the  expected 
values  associated  with  the  individual  CDFs  and  CCDFs  in  Figs.  3.11 
and  3.12.  Each  expected  value  E/i[A(10|a,  r,-) | e/u]  is  calculated  as 
indicated  in  Eq.  (3.48),  and  the  resultant  CDF  and  CCDF  are 
calculated  as  indicated  in  conjunction  with  Eq.  (3.53).  In  consis¬ 
tency  with  the  results  in  Figs.  3.11  and  3.12,  the  preceding 
calculations  use  the  samples  indicated  in  Eqs.  (3.67)  and  (3.68). 

The  estimated  expected  value  E£{E/i[A(10|a,  r)|e^]}  over  aleatory 
and  epistemic  uncertainty  is  also  shown  in  Fig.  3.13  and  corresponds 
to  the  estimated  expected  value  E£{E/i[y(t|a,eM)|e^]}  defined  in 
Eq.  (3.54).  The  quantity  Ee{E/\[A(10|a,r)|e/i]}  is  the  outcome  of 
reducing  all  the  information  in  Figs.  3.1 1  and  3.12  to  a  single  number. 

3.7.  Kaplan-Garrick  ordered  triple  representation  for  risk 

The  Kaplan-Garrick  ordered  triple  representation  for  risk  is 
introduced  in  conjunction  with  Questions  Ql,  Q2  and  Q3  in 
Section  2  as  a  way  of  intuitively  describing  risk.  More  formally, 
this  representation  characterizes  risk  as  an  ordered  triple  of 
the  form 

(Sj.pSj.cSj),  j  =  1,2 . nS,  (3.69) 

b 
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A  :  EA[A(lO\a,  die,,] 

Fig.  3.13.  Estimated  CDF  and  CCDF  (i)  for  expected  values  E„[A(10|a,  rt)|e>u],  i=l, 
2,  ....  nSE= 200,  associated  with  CDFs  and  CCDFs  in  Figs.  3.11  and  3.12  and  (ii) 
with p£(E„[a(10|a,  r)|ea]  < A}  and  pE(A < £„[.A(10!a,  r)|e„]}  used  as  mnemonics  for 
estimated  probabilities  of  the  form  p£[Wy(10)]  and  pE[W|(  10)]  defined  in  conjunc¬ 
tion  with  Eq.  (3.53). 

where  5,  is  a  set  of  similar  occurrences,  pSj  is  the  probability  of 
the  set  Sj,  cS,  is  a  vector  of  consequences  associated  with  Sj,  the 
sets  Sj  are  disjoint  (i.e.,  <S,- n Sj  =  0  for  iv=j),  and  the  set  uSj 
contains  all  risk  significant  occurrences  in  the  particular  universe 
under  consideration. 

The  representation  in  Eq.  (3.69)  is  simply  a  way  to  describe 
the  components  of  approximations  to  integrals  of  the  form 
appearing  in  Eq.  (3.48)  obtained  with  stratified  sampling  from 
the  sample  space  A  for  aleatory  uncertainty.  With  stratified 
sampling,  the  expected  value  E„[y(t[a,  eM)|e„]  and  its  defining 
integral  in  Eq.  (3.48)  are  approximated  by 

E„ty(t|a,eM)|e/i]  =  Jj/(t\a,eM)dA(a\eA)dA  s  ^y(t|aj,eM)Pa(.4j[e,i), 

(3.70) 

where  the  Aj  are  disjoint  subsets  of  A  with  uAj  =  A,  pA(Aj  e„)  is 
the  probability  of  Aj,  and  aj  is  a  representative  element  of  Aj. 
With  respect  to  the  representation  in  Eq.  (3.69),  Aj  corresponds  to 


Sj,  pA(Aj\eA)  corresponds  to  pSj,  y(t|aj,  eM)  corresponds  to  an 
element  of  cS,,  and  nSA  corresponds  to  nS. 

In  turn,  the  defining  probabilities  for  CDFs  and  CCDFs  are 
given  by 

p„[y(t|a,eM)<y|e„]^  ^  6y\y(t \ aj,eM)]pA(Aj \ eA)  (3.71) 

and 

pA\y  <y(t|a,eM)|e„]  s;  J2  Sy[y(t|aj.eM)]Pa(^i|eA),  (3.72) 

respectively. 

In  summary,  the  Kaplan-Garrick  ordered  triple  representation 
for  risk  provides  a  simple  and  intuitive  description  of  the  basic 
components  of  a  risk  assessment.  Specifically,  this  representation 
provides  a  display  of  the  answers  to  the  first  three  basic  questions 
that  underlie  a  risk  assessment:  (i)  “What  can  happen?”,  (ii)  “How 
likely  is  it  to  happen?”,  and  (iii)  “What  are  the  consequences  if  it 
does  happen?”.  However,  it  is  important  to  recognize  that  this 
representation  is  simply  a  way  of  decomposing  approximations  to 
integrals  involving  aleatory  uncertainty  into  their  basic  compo¬ 
nents  as  indicated  in  Eqs.  (3.70)-(3.72).  Use  of  the  Kaplan-Garrick 
ordered  triple  representation  for  risk  is  suggested  in  App.  A  of  the 
NAS/NRC  report  on  QMU  [77], 

3.8.  Verification  and  validation 

Verification  and  validation  are  two  very  important  components 
of  a  QMU  analysis  that  are  intimately  connected  with  the  assess¬ 
ment  and  representation  of  uncertainty,  where  (i)  verification  is 
the  process  of  determining  that  a  model  implementation  accu¬ 
rately  represents  the  developers’  conceptual  description  of 
the  model  and  the  solution  to  the  model,  and  (ii)  validation  is 
the  process  of  determining  the  degree  to  which  a  model  is  an 
accurate  representation  of  the  real  world  from  the  perspective  of 
the  intended  uses  of  the  model  (p.  3,  [107];  [108-113]). 

Sampling-based  sensitivity  analysis  as  described  in  Ref.  [56] 
and  illustrated  in  Sections  4  and  5  is  a  powerful  tool  for  checking 
for  analysis  errors  and  thus  is  an  important  component  of  analysis 
verification.  Further,  model  validation  is  an  important  contributor 
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to  the  insights  that  ultimately  lead  to  the  definition  of  the 
probability  space  that  characterizes  epistemic  uncertainty. 

Techniques  for  verification  and  validation  are  not  the  focus  of 
this  presentation  but  are  necessary  components  of  a  credible 
QMU  analysis.  The  importance  of  verification  and  validation  is 
emphasized  in  the  NAS/NRC  report  on  QMU  (p.  22,  Ref.  [77]). 

3.9.  An  Admonition 

As  the  reader  has  undoubtedly  observed,  this  section  essen¬ 
tially  presents  the  same  calculation  over  again  and  over  again  as 
different  expected  values  and  probabilities  are  calculated.  As  a 
reminder,  the  probabilities  that  define  CDFs  and  CCDFs  are 
actually  themselves  expected  values;  specifically,  these  probabil¬ 
ities  are  expected  values  for  indicator  functions  (i.e.,  functions  of 
the  form  djx)  and  <5x(x)  as  defined  in  conjunction  with  Eq.  (3.2)). 
What  is  changing  in  the  calculations  is  the  sample  space  under 
consideration  (e.g.,  £A,  £M,  £  =  £Ax  £M,  A, ...),  the  probability 
space  associated  with  the  sample  space  (e.g.,  (£A,  EA,  pFA),  (£M, 
EMI,  Pem ),  (£,  E,  pE),  {A,  A,  pA),...),  and  the  function  being 
integrated  (e.g.,  y(t|a,  eM),  8y\y(t  a,  eM)],  <5y[y(t  a,  eM)],  E„[y(t  a, 
eM)|®a],---)-  However,  at  a  conceptual  level,  the  basic  calculation 
remains  the  same.  The  calculations  are  repeated  to  be  explicit 
about  the  sample  space,  probability  space  and  function  involved 
rather  than  because  of  inherent  conceptual  differences  in  the 
probabilistic  basis  of  the  calculation. 

Now  for  the  admonition.  When  confronted  with  a  probability  or 
a  calculation  involving  probability,  the  first  two  questions  to  ask 
are  “What  is  the  sample  space?”  and  “What  subset  of  the  sample 
space  is  under  consideration?”.  If  you  do  not  know  the  answers  to 
these  two  questions,  then  you  do  not  know  enough  to  mean¬ 
ingfully  assess  the  probability  or  calculated  result  under  considera¬ 
tion.  Further,  if  the  source  of  the  probability  or  calculated  result 
cannot  supply  precise  answers  to  these  two  questions,  then  there  is 
reason  to  be  cautious  with  respect  to  the  meaning  and  correctness 
of  such  results.  Basically,  having  a  probability  without  knowing  the 
associated  sample  space  and  the  subset  of  that  sample  space  for 
which  the  probability  is  defined  is  analogous  to  knowing  the 
answer  to  a  question  without  knowing  what  the  question  is. 

For  the  preceding  reason,  this  section  has  been  very  explicit  in 
stating  the  sample  space  and  the  relevant  subsets  of  that  sample 
as  different  quantities  have  been  introduced  and  defined.  This 
results  in  some  repetition  at  a  conceptual  level  but  has  the 
positive  effect  of  unambiguously  defining  the  individual  quanti¬ 
ties  under  consideration. 


4.  QMU  with  epistemic  uncertainty:  characterization 
with  probability 

The  use  of  probability  to  represent  the  epistemic  uncertainty 
associated  with  results  of  the  form 

y(t|a,eM)=/(t|a,eM)  (4.1) 

is  extensively  discussed  in  Section  3.3,  where  y( 1 1 a,  eM)  is  a  generic 
real-valued  quantity  conditional  on  a  specific  realization  a  of 
aleatory  uncertainty  and  eM  is  a  vector  of  epistemically  uncertain 
analysis  inputs.  The  result  y(t|a,  eM)  is  epistemically  uncertain  as  a 
consequence  of  the  epistemic  uncertainty  associated  with  the 
elements  of  eM.  Given  that  the  realization  a  of  aleatory  uncertainty 
is  fixed,  analyses  related  toy(t|a,  eM)  involve  two  of  the  three  basic 
analysis  components  discussed  in  Section  3.2:  (i)  EN2,  a  model  that 
predicts  system  behavior  (i.e.,  a  function  j{t\ a,  eM)),  and  (ii)  EN3,  a 
probabilistic  characterization  of  epistemic  uncertainty  (i.e.,  a  prob¬ 
ability  space  (£M,  EM,  pr.:M)  that  characterizes  the  epistemic 
uncertainty  associated  with  the  elements  of  eM). 


Margins  can  be  defined  fory(t|a,  eM)  in  a  variety  of  ways,  and 
in  turn,  the  epistemic  uncertainty  associated  with  eM  results  in 
uncertainty  in  y(t|a,  eM)  and  the  margins  that  derive  from  y(t|a, 
eM ).  At  an  intuitive  level,  a  margin  corresponds  to  a  difference 
between  a  required  level  of  performance  and  an  estimated  level  of 
performance,  with  a  positive  margin  indicating  that  the  required 
level  of  performance  is  met  and  a  negative  margin  indicating  that 
the  required  level  of  performance  is  not  met.  Multiple  examples 
of  how  margins  could  be  defined  are  introduced  in  this  section,  in 
Section  5  and  also  in  Ref.  [80], 

This  section  uses  the  function  Q(t|a,  eM)  introduced  in  Section  3.4 
to  illustrate  a  variety  of  ways  in  which  QMU  analyses  could  arise 
and  be  carried  out  in  the  context  of  analyses  that  involve  a  generic 
result  y(t|a,  eM)  of  the  form  indicated  in  Eqs.  (3.24)  and  (4.1). 
Further, 

eM  =  [Bm\Am2Am3Am4Am5\  =  [f-.R.C.Eo,/1.]  (4.2) 

has  the  properties  defined  in  conjunction  with  Eq.  (3.32),  and  the 
corresponding  probability  space  (£M,  EM,  pEM)  that  characterizes 
the  epistemic  uncertainty  associated  with  eM  is  defined  in  conjunc¬ 
tion  with  Eqs.  (3.33)-(3.43).  The  time-dependent  behavior  of  Q(t|a, 
eM)  is  illustrated  in  Fig.  3.5. 

The  examples  presented  in  this  section  use  an  LHS 

eMi  =  [eMi,i.eM2,i . eM5,i]  =  [Li, Ri.Ci.Eoi, Ail  i=1,2,. .  .,nSE=  200, 

(4.3) 

from  £M  generated  in  consistency  with  the  distributions  that 
define  the  probability  space  (£M,  EM,  Pem).  In  turn,  evaluation  of 
Q(t|a,  eMi )  for  elements  of  the  preceding  sample  produces  a 
mapping 

[eMi,Q(t|a,eMi)],  i=  1,2 . nSE=  200,  (4.4) 

from  uncertain  analysis  inputs  to  analysis  results  that  is  used  in 
the  generation  of  the  example  results  presented  in  this  section. 

The  following  topics  related  to  QMU  in  the  presence  of  only  epi¬ 
stemic  uncertainty  are  considered  in  this  section:  epistemic  uncer¬ 
tainty  with  a  specified  bound  (Section  4.1),  epistemic  uncertainty 
with  a  specified  bounding  interval  (Section  4.2),  epistemic  uncertainty 
with  a  specified  bounding  interval  over  time  (Section  4.3),  epistemic 
uncertainty  with  an  uncertain  bound  (Section  4.4),  and  information 
loss  associated  with  a  “margin/uncertainty"  ratio  (Section  4.5).  The 
bounds  considered  in  Sections  4.1 -4.4  correspond  to  the  require¬ 
ments  that  give  rise  to  margins  (i.e.,  the  differences  between  required 
system  performance  and  predicted  system  performance). 

As  indicated  at  the  beginning  of  Section  3.3,  the  NAS/NRC 
report  on  QMU  emphasizes  the  importance  of  the  quantification 
of  the  epistemic  uncertainty  in  analysis  results  that  derives  from 
epistemic  uncertainty  in  analysis  inputs  (Recommendation  1-2, 
p.  22,  Ref.  [77]).  The  results  presented  in  Sections  4.1 -4.4 
illustrate  analyses  of  this  type. 

4.1.  Epistemic  uncertainty  with  a  specified  bound 

For  this  example,  a  fixed  bound  is  assumed  to  exist  with 
respect  to  the  value  for  Q(0.1|a,  eM).  Possibilities  include  lower 
bounds  on  Q{0.1  |a,  eM)  (e.g.,  Q*i  =  0.075  and  0*2=0.09  in  Fig.  4.1a) 
and  upper  bounds  on  Q(0.1  |a,  eM)  (e.g.,  0*3=0.105  and  Q*4=0.125 
in  Fig.  4.1b).  Consistent  with  the  nature  of  the  bounds  being 
illustrated,  the  distribution  of  possible  values  for  Q(0.1|a,  eM )  is 
summarized  with  a  CDF  in  Fig.  4.1a,  and  the  distribution  of 
possible  values  for  Q(0.1|a,  eM)  is  summarized  with  a  CCDF  in 
Fig.  4.1b.  Specifically,  the  CDF  in  Fig.  4.1a  displays  the  probability 
of  being  less  than  a  specific  bound,  which  is  the  probability  of 
interest  for  the  lower  bounds  Q*i  and  Q*2,  and  the  CCDF  in 
Fig.  4.1b  displays  the  probability  of  being  greater  than  a  specified 
bound,  which  is  the  probability  of  interest  for  the  upper  bounds 
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Fig.  4.1.  Example  bounds  on  Q(0.1|a,  eM):  (a)  lower  bounds  Qm =0.075  and  <1,2=0.09 
and  estimated  CCDF  for  Q(0.1  ja,  eM). 


Q*3  and  Q*4.  The  CDF  and  CCDF  in  Fig.  4.1  were  generated  with 
the  sample  and  associated  mapping  in  Eqs.  (4.3)  and  (4.4)  as 
described  in  conjunction  with  Eq.  (3.25). 

For  notational  simplicity,  the  ordinates  in  Fig.  4.1a  and  b  are 
assigned  the  labels  “Cumulative  Probability”  and  “Complementary 
Cumulative  Probability”  rather  than  the  more  explicit  but  also 
more  complex  labeling  used  with  CDFs  and  CCDFs  in  Section  3. 
This  labeling  convention  is  also  used  with  other  similar  Figures. 

All  sampled  values  for  Q(0.1  |a,  eMi )  are  above  the  bound  Q*,. 
However,  this  is  not  the  case  for  Q*2,  with  the  CDF  in  Fig.  4.1a 
indicating  that  the  probability  of  Q(0.1|a,  eM)  being  below 
0*2=0.09  is  approximately  0.200. 

All  sampled  values  for  Q(0.1  a,  eM<)  are  below  the  bound  Qm. 
However,  this  is  not  the  case  for  Q*3,  with  the  CCDF  in  Fig.  4.1b 
indicating  that  the  probability  of  Q(0.1|a,  eM)  being  above 
0*3=0.105  is  approximately  0.155. 

The  margins  between  Q(0.1  |a,  eM)  and  the  bounds  Q*fc,  k=  1,  2, 
3,  4,  indicated  in  Fig.  4.1  are  defined  by 


Qmfc(01  |a,eM)  = 


0.(0. 1  |a,eM)-Q*k  fork  =1,2 
Q*k-Q(0.1|a,eM)  fork  =  3,4, 


(4.5) 


with  Qmfc(0.1 1  a,  eM)  >  0  indicating  that  a  specified  bound  is  satisfied 
and  QmtjO.I  a,  eM)<0  indicating  that  a  specified  bound  is  not 
satisfied  (i.e.,  a  positive  margin  is  good  and  a  negative  margin 
is  bad).  As  a  result  of  0(0.1  |a,  eM)  being  epistemically  uncertain, 
the  corresponding  margins  0^(0. 1  a,  eM),  fc=l,  2,  3,  4,  are  also 
epistemically  uncertain  and  have  an  uncertainty  structure  that 
derives  from  the  uncertainty  structure  assumed  for  eM  (Fig.  4.2). 
Representations  of  the  form  shown  in  Fig.  4.2  provide  a  complete 
display  of  the  uncertainty  associated  with  the  margins  Qm^O.l  |a, 
eM),  k=l,  2,  3,  4,  and  thus  a  complete  QMU  representation  of 
margin  uncertainty. 

An  alternative  format  involves  the  use  of  normalized  margins 
defined  by 

Qjik(0.1  |a,eM)  =  Qmfc(0  1  |a,eM)/Q*k 

_  J  [0,(0. 1  |a,eM)-Q*k]/Q*fc  for  k=  1,2 
“  \  [Qftfc-Q(0.1 1 a ,eM)]/(hk  for  k  =  3,4,  (4  b) 


which  expresses  margin  as  a  fraction  of  the  corresponding  bounding 
value  (Fig.  4.3).  This  format  has  the  advantage  in  that  it  presents 
margin  as  a  multiple  of  the  bounding  value,  which  is  a  presentation 
format  that  some  individuals  prefer.  However,  it  has  the  disadvantage 
that  it  does  not  present  the  actual  size  of  the  margin. 


It  is  sometimes  stated  that  QMU  corresponds  to  the  determina¬ 
tion  of  the  ratio  “margin/uncertainty.”  Unfortunately,  it  is  not 
always  apparent  how  this  imagined  concept  translates  into  quan¬ 
tities  that  are  mathematically  defined  and  conceptually  useful.  In 
contrast,  margin  results  of  the  form  illustrated  in  Fig.  4.2  are 
mathematically  well-defined,  computationally  practicable,  and 
meaningful  in  a  decision  context  as  all  available  information  about 
margins  and  their  associated  uncertainty  is  presented. 

Two  possible  definitions  of  “margin/uncertainty"  for  an  arbi¬ 
trary  margin  Qm(t|a,  eM)  (e.g.,  Qnfc(0.1  |a,  eM)  for  k=l,  2, 3  or  4)  are 


Qm/u(f  |a,ejvr)  = 


_ Om,o.5(f|a.eM) _ 

Qm,o.5(t|a,eM)-Qmi0.o5(f|a.eM) 


(4.7) 


Qm/u(t|a,eM)  = 


Qm(t|a,eM) 

Qm(t|a,eM)-Qm,0.05(t|a,eM)' 


(4.8) 


where  (i)  Om,o.5(t|a,eM)  is  the  median  (i.e.,  q=  0.5  quantile)  for  Qm 
(t  | a,  eM)J_(ii)  Qm.o.osQ  a,eM)  is  the  0.05  quantile  for  Qm(t|a,  eM), 
and  (iii)  Qm(t|a,eM)  is  the  expected  value  for  Qm(t|a,  eM). 

As  illustrated  in  Fig.  4.2,  quantities  such  as  0m,o.5(t|a,  eM), 
0m,o.o5(t|a,  eM)  and  Qm(t|a,  eM)  are  typically  estimated  with 
sampling-based  procedures.  With  respect  to  the  more  detailed 
notation  used  in  Sections  3.3  and  3.4,  Qm,o.5(t|a,  eM)  and  Qm,o.o5(t|a, 
eM)  correspond  to  Qj;Mt/[Qm(f|a,  eM)]  for  q=0.5  and  0.05,  respec¬ 
tively,  and  Qm(t|a,  eM)  corresponds  to  £EM[Qm(t|a,  eM)]. 

The  quantities  Qm/^tja,  eM)  and  Qm/„(t|a,  eM )  defined  in 
Eqs.  (4.7)  and  (4.8)  are  based  on  using  the  median  and  mean 
margins  Qm.o.sjf  a,  eM)  and  Qm(t|a,eM)  as  best  estimates  for  an 
uncertain  margin  and  then  defining  uncertainty  as  the  difference 
between  this  best  estimate  and  a  low  quantile  (e.g.,  q=0.05)  of 
the  uncertainty  distribution  for  margin.  In  general,  large  positive 
margins  are  good  and  small  or  negative  margins  are  bad;  in  turn, 
margins  associated  with  small  quantiles  correspond  to  small  differ¬ 
ences  between  required  bounds  and  predicted  system  behavior  and 
thus  are  less  desirable  than  margins  associated  with  larger  quantiles. 
As  a  result,  the  differences  in  the  denominators  in  Eqs.  (4.7)  and  (4.8) 
provides  a  measure  of  the  epistemic  uncertainty  present  in  the 
determination  of  the  margin  under  consideration. 

At  least  notionally,  values  for  Qm/u(t  a,  eM)  and  Qm/u(t|a,  eM) 
significantly  larger  than  1  are  good  because  this  situation  results 
when  Qm,o.5(tJa,  eM)  and  Qm(t|a,  eM)  are  close  to  Qn.o.osjt  a,  eM) 
in  value,  which  in  turn  implies  that  there  is  little  epistemic 
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uncertainty  present  in  the  estimation  of  the  margin  under  con¬ 
sideration.  However,  values  for  Qjn/U(t|a,  eM)  and  Q.m/U(fla,  eM) 
significantly  greater  than  1  do  not  exclude  the  undesirable 
situation  in  which  the  estimated  margins  are  very  close  to  0. 
Values  for  Qm/U(t|a,  eM)  and  Q,m/U(t|a,  eM)  equal  to  or  only  slightly 
larger  than  1  are  undesirable  because  this  situation  results  when 
0m,o.o5(t|a,  eM)  is  equal  to  or  only  slightly  larger  than  0,  and  values 
for  Qm/U(t|a,  eM)  and  Qm/U(t|a,  eM)  less  than  1  are  bad  because  this 
situation  results  when  Qm.o.osCt  a,  eM)  is  negative.  It  is  important 
to  recognize  that  very  different  distributions  for  Qm(t|  a,  eM )  can 
result  in  similar  values  for  Qm/u(t|a,  eM)  and  also  for  Qm/U(t|a,  eM). 
As  a  result,  consideration  of  only  summary  values  such  as 
C2m/u(t|a,  eM)  and  Q.m/U(t|a,  eM)  can  result  in  an  incomplete  and 
potentially  misleading  assessment  of  the  implications  of  the 
uncertainty  associated  with  the  margin  Qm(t|a,  eM).  Additional 
discussion  of  the  nature  of  “margin/uncertainty”  results  is  provided 
in  Section  4.5. 

For  the  example  margins  under  consideration  in  this  section 
(Fig.  4.2)  and  the  normalization  defined  in  Eq.  (4.7),  the  values  for 
Qm/u(  0.1 1  a,  eM)  are 

Q^.KO.l  |a,eM)  =  0.022/(0.022-0.009)  =  1.7,  (4.9) 

(4.10) 


Qm/u,3(0.1  |a,eM)  =  0.0080/[0.0080—(— 0.0068)]  =  0.54,  (4.11) 

and 

Qm/u, 4(0.1  |a,eM)  =  0.028/(0.028-0.013)  =  1.9.  (4.12) 

The  values  for  Ctm/Uifc(0.1  |a,eM)  are  essentially  the  same  as  the 
values  for  CW^fO.l  |a,  eM)  in  Eqs.  (4.9)-(4.12)  because  of  the 
similarity  of  the  mean  and  median  values  for  Qmk(0.1  |a,  eM)  (see 
Fig.  4.2).  However,  such  similarity  will  not  exist  in  many  analyses. 

As  discussed  in  Section  4.5,  “margin/uncertainty”  ratios  of 
the  form  defined  in  Eqs.  (4.7)  and  (4.8)  and  illustrated  in 
Eqs.  (4.9)-(4.12)  are  in  (i)  the  interval  [1,  +  oo)  if  the  best  margin 
estimate  (e.g.,  the  mean  or  median)  is  positive  and  the  lower 
margin  estimate  (e.g.,  the  0.05  quantile)  is  nonnegative,  (ii)  the 
interval  [0,  1 )  if  the  best  margin  estimate  is  nonnegative  and  the 
lower  margin  estimate  is  negative,  and  (iii)  the  interval  (-  oo ,  0)  if 
the  best  margin  estimate  and  the  lower  margin  estimate  are  both 
negative.  With  respect  to  the  preceding  statements,  it  is  tacitly 
assumed  that  the  best  margin  estimate  is  greater  than  the  lower 
margin  estimate.  Further,  the  indicated  ratio  (i)  equals  1  only 
when  the  best  estimate  is  positive  and  the  lower  estimate  is  0, 
(ii)  equals  0  only  when  the  best  estimate  is  0  and  the  lower 
estimate  is  negative,  and  (iii)  is  undefined  when  the  best  estimate 


Qm/uii  0.1  |a,eM)  =  0.0070/[0.0070-(— 0.0056)]  =  0.56, 
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and  the  lower  estimate  are  equal.  Consistent  with  the  indicated 
relationships,  the  “margin/uncertainty”  ratios  Qm/u,i(0.1  |a,  eM)= 
1.7  and  Qm/U,4(0.1  |a,  eM)=1.9  in  Eqs.  (4.9)-(4.12)  are  greater 
than  1  because  both  the  best  and  lower  margin  estimates  are 
nonnegative,  and  the  “margin/uncertainty”  ratios  Qm/u,2(0.1  |a, 
eM)=0.56  and  Qm/u.aCO.l  |a,  eM)=0.54  in  Eqs.  (4.10)  and  (4.11) 
are  in  the  interval  (0,  1)  because  the  best  and  lower  margin 
estimates  are  positive  and  negative,  respectively. 

The  “margin/uncertainty”  results  defined  in  Eqs.  (4.7)  and  (4.8) 
and  illustrated  in  Eqs.  (4.9)-(4.12)  reduce  the  individual  CDFs  in 
Fig.  4.2  to  single  numbers.  As  a  result,  a  large  amount  of 
information  is  lost  in  this  reduction.  Further,  as  discussed  and 
illustrated  in  Section  4.5,  a  “margin/uncertainty”  ratio  provides 
no  information  on  the  actual  values  for  the  best  and  lower  margin 
values  used  in  the  determination  of  this  ratio.  Thus,  for  example, 
there  is  no  way  to  use  the  results  in  Eqs.  (4.9)-(4.12)  to  retrieve 
the  margin  values  used  in  the  determination  of  these  results. 
Simply  put,  all  the  information  in  Figs.  4.1  and  4.2  has  been  lost. 

At  their  most  extreme,  Qm/U(t|a,  eM)  and  Q.m/U(t|a,  eM)  have  the 
forms 


and 


Qm/u(t|a,eM)  = 


Qm(t|a,eM) 

QmCtla.eMl-Qm^.ooftla.eM) 


where 


(4.14) 


Om,o.oo(t|a,eM)  =  0.00  quantile  for  Qm(t  a,eM) 

=  inf{Qm(t|a,eM) :  eMe£}. 

In  words,  Qm,o.oo(t|a,  eM)  is  the  smallest  possible  value  for  the 
margin  Qm^a,  eM).  As  a  result  of  the  inequality 
Qm,o.oo(t|a,eM)  <  Qm.o.o5(t|a,eM),  (4.15) 

use  of  Qm,o.oo(t|a,  eM)  in  the  definition  of  Qm/u(t  a,  eM)  and 
Q.m/u(f|a-  eivf)  results  in  smaller  values  for  these  quantities  than 
the  use  of  Qm,0.05(t|a,  eM). 

For  the  example  margins  under  consideration  in  this  section, 
the  values  for  Qm/U(0.1  |a,  eM)  obtained  with  Qm,o.oo(0.1  |a,  eM)  as 
indicated  in  Eq.  (4.13)  are 

Qm/u,i(0.1  a,eM)  =  0.022/(0.022-0.004)  =  1.2,  (4.16) 


Qm/u(t|a,eM)  = 


Qm.o.5(t|a,eM) 

Qm,  o.5(t  |  a,eM)-Qm,o.oo(t  |  a,eM) 


Qm/ujCO.l  |a,eM)  =  0.0070/[0.0070-(-0.0106)]  =  0.40,  (4.17) 

Qm^fO.l  |a,eM)  =  0.0080/[0.0080-(-0.0153)]  =  0.34, 


(4.18) 
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and 

Qm/uA(0.\  |a,eM)  =  0.028/(0.028-0.005)  =  1.2.  (4.19) 

Again,  the  values  for  Qm/U  k(0. 1  a,eM)  defined  in  Eq.  (4.14)  are  very 
similar  to  the  values  for  Qm/ii,fc(0-l  |a.  eM)  defined  in  Eq.  (4.13) 
because  of  the  similarity  of  the  mean  and  median  values  for 
Qmk(0.1|a,  eM).  As  noted  in  conjunction  with  the  inequality  in 
Eq.  (4.15),  “margin/uncertainty”  ratios  obtained  with  Qm,o.oo(0.1  |a,  eM) 
are  smaller  than  the  ratios  obtained  with  CWostO.l  |a,  eM)  (i.e., 
compare  results  in  Eqs.  (4.9)-(4.12)  with  results  in  Eqs.  (4.16)-(4.19)). 

The  importance  of  sensitivity  analysis  is  recognized  in  the 
NAS/NRC  report  on  QMU  (pp.  14-15,  Ref.  [77]).  Indeed,  sensitivity 
analysis  should  be  an  integral  part  of  any  QMU  analysis.  As  an 
example,  a  sensitivity  analysis  for  Q(0.1  a,  eM)  based  on  stepwise 
regression  analysis  is  presented  in  Table  4.1.  Specifically,  stepwise 
regression  analysis  is  used  to  explore  the  mapping 

[eMi.Q(0.1|a,eMi)],  i=l,2 . nSE=  200,  (4.20) 

used  to  generate  the  uncertainty  results  in  Figs.  4.1 -4.3.  With  this 
procedure,  variable  importance  is  indicated  by  the  order  in  which 
variables  are  selected  in  the  stepwise  process,  the  incremental 
changes  in  R2  values  with  the  entry  of  individual  variables  into 
the  regression  model,  and  the  sizes  and  signs  of  the  standardized 
regression  coefficients  (SRCs)  in  the  final  regression  model  (see 
Ref.  [56]  for  additional  discussion  of  regression-based  sensitivity 
analysis). 

As  examination  of  Table  4.1  shows,  the  dominant  variables 
affecting  the  uncertainty  in  Q(0.1  a,  eM)  are  £0  and  C.  Specifically, 
the  positive  SRCs  associated  with  £0  and  C  indicate  that  Q(0.1  |a,  eM) 
tends  to  increase  in  value  as  each  of  these  variables  increases. 


In  addition,  small  negative  effects  are  indicated  for  R  and  X,  and  a 
small  positive  effect  is  indicated  for  L. 

The  examination  of  scatterplots  is  also  an  informative  part  of 
sampling-based  sensitivity  analysis.  For  example,  the  scatterplots  in 
Fig.  4.4  clearly  reveal  the  positive  effects  of  E0  and  C  on  Q(0.1  |a,  eM) 
and  the  resultant  outcomes  that  negative  or  small  positive  margins 
associated  with  requirements  Qm  and  Qt2  occur  for  small  values  of 
E0  and  C  and  (ii)  negative  or  small  positive  margins  associated  with 
requirements  Qj, 3  and  Qm  occur  for  large  values  of  E0  and  C. 

Regression-based  sensitivity  analysis  could  also  be  carried  out 
for  the  margins  Qmk(0.1  a,  eM),  k=  1,  2,  3,  4,  defined  in  Eq.  (4.5) 
and  illustrated  in  Fig.  4.2.  However,  given  that  the  margins  are 
simply  affine  transformations  (i.e.,  linear  scalings  plus  constant 
shifts)  of  Q(0.1 1 a,  eM)  defined  by  the  bounds  Qm,  k=l,  2,  3,  4,  the 
resultant  regression  analyses  for  Qmk(0.1  |a,  eM),  k=  1,  2,  would  be 
the  same  as  presented  in  Table  4.1  as  a  result  of  the  defining 
transformation 

Qmk(01  |a,eM)  =  Q(0.1  la^-Owc  (4.21) 

for  k=  1,  2,  and  the  resultant  regression  analyses  for  Qmi/0.1  a, 
eM),  k= 3,  4,  would  also  be  the  same  as  presented  in  Table  4.1 
except  for  a  reversal  in  the  signs  of  the  SRCs  as  a  result  of  the 
defining  transformation 

Qmk(01  |a,eM)  =  Qm-Q(0.1  |a,eM)  (4.22) 

for  k= 3,  4.  Similarly,  the  scatterplots  for  the  margins  Qmi/0.1  a, 
eM),  fc=l,  2,  3,  4,  would  effectively  convey  the  same  information 
as  the  scatterplots  for  Q(0.1  a,  eM)  in  Fig.  4.4. 

4.2.  Epistemic  uncertainty  with  a  specified  bounding  interval 


Table  4.1 

Stepwise  regression  analysis  to  identify  uncertain  variables  affecting  Q(0.1  a,  eM). 


Step3 * * * *  Variable11  SRCC  R2'1 


0.70  0.51 

0.63  0.91 

-0.22  0.96 

-0.12  0.98 

0.06  0.98 


3  Steps  in  stepwise  regression  analysis  with  an  a-value  of  0.01  or  less  required 

for  a  variable  to  enter  a  regression  model. 

b  Variables  listed  in  the  order  of  selection  in  regression  analysis. 

c  SRCs  for  variables  in  final  regression  model. 

d  Cumulative  R2  value  with  entry  of  each  variable  into  regression  model. 


A  QMU  problem  involving  a  bounding  interval  rather  than 
simply  an  upper  or  lower  bound  is  now  considered.  Specifically, 
the  problem  involves  a  specified  interval  within  which  the 
quantity  of  interest  is  required  to  be  located.  For  the  quantity 
0(0.1  |a,  eM),  this  involves  the  specification  of  an  interval  [Qh,Qb] 
such  that  the  inequalities 

Q^Q^.lja,  eM)<Q6  (4.23) 

hold  (Fig.  4.5).  For  illustration,  [Q^.Qi,]  is  assumed  to  equal  [0.08, 
0.12]  as  indicated  in  Fig.  4.5.  This  example  corresponds  to  considera¬ 
tion  of  what  is  called  a  “gate”  in  some  discussions  of  QMU  [1,4], 
There  are  several  ways  in  which  the  epistemic  uncertainty 
associated  with  compliance  with  the  specified  bounds  can  be 


Fig.  4.4.  Scatterplots  for  Q(0.1  |a,  eM):  (a)  [Eoi,  0(0.1  |a,  eMi)],  i=l,2 . nS£=200,  and  (b)  [Q,  Q(0.1  |a,  eMi)],  i=l,2 . nS£=200. 
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Q  (0.1  la,  eM) 


Fig.  4.5.  Example  bounding  interval  [Q6,  Qb]=[0.08,  0.12]  for  Q(0.1  a,  eM). 


represented.  The  easiest  is  simply  to  consider  whether  or  not 
Q(0.1|a,  eM)  falls  within  the  specified  bounds.  This  involves 
consideration  of  the  indicator  function 


<5[Q(0.1|a,  eM)]  = 


1  if  Qj,  <  0.(0. 1 
0  otherwise 


and  the  associated  sets 

X+  ={eM:eMe£M  and  5[Q(0.1  |a,eM)]  =  1} 


and 

X~  =  {eM-.eMe£M  and  <5[Q(0.1  a,eM)]  =  0}.  (4.26) 

Then,  the  probabilities  of  compliance  and  noncompliance  are 
given  by 


Pem(*+)  =  £  <5[O(0.1 1  a ,eM)/nSE]  =  0.985 


Pem(A-)  =  1  -Pem(*+)  =  1-0.985  =  0.015, 


(4.27) 


(4.28) 


respectively. 

The  representation  in  the  preceding  paragraph  summarizes 
the  uncertainty  in  whether  or  not  the  specified  interval  bound 
will  be  satisfied.  However,  the  uncertainty  in  the  location  of 
0(0.1 1  a,  eM)  relative  to  the  ends  of  the  bounding  interval  [Qft,  Oft] 
is  not  indicated.  The  consideration  of  this  uncertainty  requires  the 
determination  of  margins  and  the  uncertainty  associated  with 
these  margins.  Specifically,  a  margin  associated  with  the  contain¬ 
ment  of  Q(0.1 1  a,  eM)  in  the  interval  [Qh,Oft]  can  be  defined  by 


Qm(0.1  |a,eM)  = 


0.(0. 1  |a,eM)-Qh 
Oft— Q(0.1  |a,eM), 


(4.29) 


with  the  result  that  (i)  On(0.1  |a,  eM)  is  nonnegative  if  Q(0.1  |a,  eM) 
falls  within  the  interval  [Q6,Oh].  and  (ii)  On(0.1  |a,  eM)  is  negative 
if  0(0.1  |a,  eM)  falls  outside  the  interval  [Oh,  Os]-  In  turn,  Qn,(0.1 1 a, 
eM)  has  an  uncertainty  structure  that  derives  from  the  uncertainty 
structure  imposed  on  eM  (Fig.  4.6).  The  probability  Pem(X~)  = 
0.015  in  Eq.  (4.28)  corresponds  to  the  cumulative  probability 
associated  with  Qm(0.1  a,  eM)=0  in  Fig.  4.6. 

An  alternate  representation  is  to  use  normalized  margins. 
Specifically,  the  margin  Qm(0.1|a,  eM)  defined  in  Eq.  (4.29)  can 


Om  (0.1  |a,  eM) 


Fig.  4.6.  Estimated  CDF  summarizing  uncertainty  in  margin  Qm(0.1  a,  eM)  defined 
in  Eq.  (4.29)  for  bounding  interval  [Qh,  Qb] = [0-08,  0.12], 


On  (0.1  |a,  eM) 


Fig.  4.7.  Estimated  CDF  summarizing  uncertainty  in  normalized  margin  Q„(0.1  a,  ei«) 
defined  in  Eq.  (4.30)  for  bounding  interval  (Qfj,  Qh  ]— [0.08,  0.12], 


be  replaced  by  a  normalized  margin  Oi(0.1  |a,  eM)  defined  by 


0.(0.!  |a,eM)  =  min 


[0(0.1  la.eMl-OJ/Q,, 
[Qft— Q(0.1  |a,eM)]/Q„, 


(4.30) 


which  expresses  margin  as  a  fraction  of  the  bounding  value  from 
which  Q(0.1 1 a,  eM)  has  the  smallest  fractional  deviation  (Fig.  4.7). 

If  desired,  “margin/uncertainty”  summary  measures  of  the 
form  defined  in  Eqs.  (4.7),  (4.8),  (4.13)  and  (4.14)  can  be  defined 
for  the  distribution  of  Qn,(0.1  |a,  eM)  in  Fig.  4.6.  Specifically, 

n  ^  On,o.5(f|a,eM) 

lu  '  M  Qm,o.5(t|a.eM)-Qm,q(t|a,eM) 

_  f0.013/(0.013-0.003)  =  1.30  for  q  =  0.05 
—  \0.013/[0.013 — ( — 0.001)]  =  0.93  for  q  =  0.00 

(4.31) 

for  t=0.1  s,  and 


Om/u(t|a,eM)  = 


Qm(t|a,eM)-Qm,9(t|a,eM) 


(4.32) 


effectively  has  the  same  values  as  Qm/U(t|a,  eM)  for  t=  0.1  s 
because  of  the  similarity  of  the  mean  and  median  values  for 
Qm(0.1  a,  eM)  (see  Fig.  4.6).  However,  as  previously  discussed  in 
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Section  4.1,  a  significant  amount  of  information  is  lost  when  the 
results  in  Figs.  4.5  and  4.6  are  reduced  to  a  single  number  (see 
Section  4.5  for  additional  discussion). 

The  results  of  a  sensitivity  analysis  for  Q{0.1  a,  eM)  are  presented 
in  Table  4.1  and  Fig.  4.4.  Because  (^(O.l  |a,  eM)  as  defined  in 
Eq.  (4.29)  for  bounding  interval  e  =  [eA,eM],  is  not  an  affine 
transformation  of  Q(0.1  |a,  eM),  these  analyses  do  not  reveal  the  full 
effects  of  the  elements  of  eM  on  Qm(0.1  |a,  eM).  To  determine  these 
effects,  a  stepwise  regression  analysis  (Table  4.2)  is  initially  per¬ 
formed  for  the  mapping 

[eMi,  Qm(0. 1 1  a.ejvfi)],  f=  1,2,.  ..,nSE=  200,  (4.33) 

and  then  followed  by  an  examination  of  scatterplots. 

The  regression  analysis  in  Table  4.2  for  (^(O.l  |  a,  eM)  is  very 
poor,  with  the  final  regression  model  containing  E0  and  C  having 
an  R1 2  value  of  only  0.16.  As  a  reminder,  E0  and  C  are  the  dominant 
variables  affecting  the  uncertainty  in  Q(0.1|a,  eM)  (see  Table  4.1 
and  Fig.  4.4).  Given  the  effects  of  E0  and  C  on  Q(0.1|a,  eM), 
a  natural  next  step  is  to  examine  the  scatterplots  for  E0,  C  and 
C2in(0.1  |a,  eM)  (Fig.  4.8).  Specifically,  the  scatterplots  in  Fig.  4.8 
show  that  small  values  for  Qm(0.1  |a,  eM)  are  associated  with  both 
small  and  large  values  for  E0  and  C.  This  is  consistent  with  the 
monotonic  effects  of  E0  and  C  on  Q(0.1  a,  eM)  shown  in  the 
scatterplots  in  Fig.  4.4  and  the  dependence  of  the  margin 
C2in(0.1 1 a,  eM)  on  both  small  and  large  values  for  Q(0.1|a,  eM) 
(see  definition  of  Qm(0.1  |a,  eM )  in  Eq.  (4.29)).  Given  the  monotonic 
effects  of  E0  and  C  on  Q(0.1  |a,  eM)  and  the  definition  of  Qm(  0.1  |a, 
eM),  small  values  for  Qm(0.1  |a,  eM)  will  tend  to  occur  when  either 
(i)  both  E0  and  C  are  at  the  lower  ends  of  their  ranges  or  (ii)  both 
E0  and  C  are  at  the  upper  ends  of  their  ranges. 

The  regression  analysis  for  Qm(0.1  |a,  eM)  in  Table  4.2  fails 
because  of  the  nonmonotonic  relationships  involving  E0,  C  and 
Qm(0.1|a,  eM)  shown  in  Fig.  4.8.  Given  the  complexity  of  the 
relationships  involving  £0,  C  and  Qm(0.1|a,  eM),  a  successful 

Table  4.2 

Stepwise  regression  analysis  to  identify  uncertain  variables  affecting  the  margin 
Qm(0.1 1 a,  e„)  defined  in  Eq.  (4.29). 

Step3  Variable  SRC  it2 


1  E0  0.28  0.08 

2  C  0.27  0.16 


3  Table  structure  same  as  in  Table  4.1. 


900  950  1000  1050  1100 


regression-based  sensitivity  analysis  for  Qm(0.1|a,  eM)  would 
require  the  use  of  nonparametric  regression  procedures  [114,115]. 

As  indicated  by  this  example,  sensitivity  analyses  associated 
with  margins  defined  for  bounding  intervals  (i.e.,  gates)  can  be 
challenging.  This  can  happen  for  at  least  two  reasons.  First, 
different  subranges  of  a  variable  can  affect  compliance  with  upper 
and  lower  bounds.  Second,  different  variables  can  affect  compli¬ 
ance  with  upper  and  lower  bounds.  The  outcome  of  these  two 
effects  can  be  complex  relational  patterns  between  margins  and 
uncertain  analysis  inputs  whose  identification  requires  sophisti¬ 
cated  sensivity  analysis  procedures  (e.g.,  [114-116]). 


4.3.  Epistemic  uncertainty  with  a  specified  bounding  interval  over  time 

A  QMU  problem  involving  a  bounding  interval  at  a  fixed  point 
in  time  is  considered  in  Section  4.2.  This  problem  is  now 
increased  in  complexity  by  considering  a  situation  in  which  a 
bounding  interval  [Q.ft,  Qj,]  is  specified  for  a  quantity  such  as 
Q(t|a,  eM)  that  takes  on  values  over  a  time  interval  [tmn,  tmx] 
(Fig.  4.9).  Specifically,  the  requirement  is  that  the  values  for  Q(t|a, 
eM)  stay  within  the  bounding  interval  [Q^,  Qb]  for  tmn<t<  tmx 
(e.g.,  [Qj.Qb]  =  [0.07,0.14],  tmn =0.02  s  and  tmx=0.18  s  in  Fig.  4.9). 


t :  Time  (s) 

Fig.  4.9.  Example  bounding  interval  [Q^,  Qj,]=[0.07,  0.14]  over  the  time  interval 
[fi™,  tmx]  =  [0.02,  0.18  s]  for  Q(t|a,  eM). 


b 


Fig.  4.8.  Scatterplots  for  margin  Qm(0.1 1 a,  eM)  defined  in  Eq.  (4.29):  (a)  [Eo,-,Qm(0.1  |a,  eM)],  i=l,2 . nSE= 200,  and  (b)  [Q,  Qm(0.1|a,  eM()],  1=1,2 . nSE= 200. 
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Formally  stated,  the  requirement  is  that  the  inequalities 

Qb  <  Q(t|a,eM)  sQ*  (4.34) 

be  satisfied  for  eM  e SM  and  tmn<t<  tmx. 

Uncertainty  in  compliance  with  the  indicated  requirement  can 
be  represented  with  use  of  the  indicator  function 
<5[Q(t|a,eM):tmn<t<t™] 

=  |  i  if  Qj,  <  Q(t | a,eM)  <  Qb  for  tm„<,t<  tmx  (4  35) 
1 0  otherwise 
and  the  associated  sets 

*+  =  |  eM  :  eM  e  SM  and  c5[Q(t|a,  eM) :  tm„  <  t  <  tmx]  =  1 1 

(4.36) 

X~  =  {  eM  :  eM  e  SM  and  <5[Q(t|a,  eM) :  tm  <,  t  <,  tm]  =  0  }.  (4.37) 

Then,  the  probabilities  of  compliance  and  noncompliance  are 
given  by 


Pem(X+ )  =  2  «5[Q(t|a,eMi) :  tmn  <  t<  tmx]/nSE=  0.82  (4.38) 


and 


Pem(X~ )  =  1  -Pem(*+)  =  1-0.82  =  0.18,  (4.39) 

respectively. 

The  preceding  representation  summarizes  the  uncertainty  in 
whether  or  not  compliance  with  the  specified  bounding  interval 
over  time  will  be  satisfied.  However,  this  representation  does  not 
display  the  associated  margins.  These  margins  can  be  defined  by 


Qm(f  |a,ejvr,[tmn,tmx]) 


f  Qmn(t|a,eM,[tm„,tmx])-QA 
{  a()-Qmx(t|a,eM,[tm„,tmx]), 


(4.40) 


where 

Qmn(t|a,eM,[tmn,tmx])  =  min{Q(t|a,eM) :  tmn<t<tmx) 
and 

Qmx(t|a,eM,[tmn,tmx])  =  max{Q(t  a,eM) :  tmn  <  t<  t^}. 

In  turn,  QmCtja,  eM,  [tmn,  fmx])  has  an  uncertainty  structure  that 
derives  from  the  uncertainty  structure  imposed  on  eM  (Fig.  4.10). 


Qm(f|a,  eM,[fm„,U) 


Fig.  4.10.  Estimated  CDF  summarizing  uncertainty  in  margin  Qm(t|a,  eM,  [tm„,  tTO]) 
defined  in  Eq.  (4.40)  for  bounding  interval  [Q^,  Qb]= [0.07,  0.14]  and  time  interval 
[tmn,  ^1  =  10.02,  0.18  s], 


The  probability  Pem(X^)  =  0.18  in  Eq.  (4.39)  corresponds  to  the 
cumulative  probability  associated  with  Qm(0.1  |a,  eM,  [tmn,  tmx])=0 
in  Fig.  4.10. 

An  alternative  representation  is  to  use  normalized  margins. 
Specifically,  the  margin  Qmttja,  eM,  [tmn,  tmx])  defined  in  Eq.  (4.40) 
can  be  replaced  by  a  normalized  margin  CWt  a,  eM,  [tmn.  tmx]) 
defined  by 

Qmn(t|a.eM,[tmn,tmx])-QJ, 

-  (4.41) 

Qi,-Qmx(f|a,eM,[tmn,tmx]) 

Qb 

which  expresses  margin  as  a  fraction  of  the  bounding  value  from 
which  Q(t|  a,  eM)  has  the  smallest  fractional  deviation  (Fig.  4.11). 

If  desired,  “margin/uncertainty”  summary  measures  of  the  form 
defined  in  Eqs.  (4.7),  (4.8),  (4.13)  and  (4.14)  can  be  defined  for  the 
distribution  of  Q{0.1  a,  eM,  [tmn.  tmx])  in  Fig.  4.10.  Specifically, 
Qm/u(t|a,eM,[tm„,tnH]) 

_ _ Qm.0.5(t|a,eM,[tmn.tmx]) _ 

Qm,o.5(t|a,eM,[tmn,tmx])-Qm,„(t|a,eM,[tmn,tmx]) 

0.0082 /[0.0082—(— 0.0094)]  =  0.47  for  q  =  0.05 
0.0082/[0.0082—(— 0.0257)]  =  0.24  for  q  =  0.00  (442) 

and 

Qm/u(t|a,eM,[tmn,tmx]) 

Qm(t|a,eM,[tmn>tmx]) _ 

Q.m(  1 1  a, eM ,  [ tmn ,  tmx]) -  Qm,, ( 1 1  a, eM ,  [ tmn ,  tmx]) 

0.0069/[0.0069—(— 0.0094)]  =  0.42  for  q  =  0.05 
0.0069/[0.0069-(-0.0257)]  =  0.21  for  q  =  0.00. 

(4.43) 

However,  as  is  always  the  case,  a  significant  amount  of  information 
is  lost  when  the  results  in  Figs.  4.9  and  4.10  are  reduced  to  a  single 
number  (see  Section  4.5  for  additional  discussion). 

Sensitivity  analysis  can  also  be  performed  for  Q(t|a,  eM)  and 
the  results  summarized  by  Qn,(t|a,  eM.  [tmn.  tmxD-  A  natural 
starting  point  is  to  investigate  the  variables  affecting  Q(t|a,  eM) 
over  the  time  interval  [tmn.  tmx]  (Fig.  4.12).  To  this  end,  partial 
correlation  coefficients  (PCCs)  and  SRCs  for  Q(t |a,  eM)  and  the 
elements  of  eM  are  presented  in  Fig.  4.12. 

Related,  but  not  identical  information  is  provided  by  PCCs  and 
SRCs.  Specifically,  a  PCC  provides  a  measure  of  the  strength  of  the 


Qn(t|a,eM,[tmn,tmx])  =  n 
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linear  relationship  between  two  variables  after  the  linear  effects 
of  all  other  variables  have  been  removed,  and  a  SRC  provides  a 
measure  of  the  fraction  of  the  uncertainty  in  a  dependent  variable 
that  can  be  accounted  for  by  the  independent  variable  under 
consideration  (see  Refs.  [53,56]  for  additional  discussion  of  PCCs 
and  SRCs).  Although  their  numeric  values  differ,  the  absolute 
values  of  PCCs  and  SRCs  provide  the  same  orderings  of  variable 


importance  when  no  correlations  between  the  independent  vari¬ 
ables  (i.e.,  the  elements  of  eM)  are  present.  For  comparison,  both 
PCCs  and  SRCs  are  presented  in  Fig.  4.12.  For  presentation 
purposes,  PCCs  can  be  preferable  to  SRCs  because  PCCs  tend  to 
be  more  spread  out  in  the  interval  [-1,  1]  than  SRCs,  with  the 
result  that  a  single  plot  frame  containing  multiple  time-depen- 
dent  PCCs  is  easier  to  read  than  a  single  plot  frame  containing 
multiple  time-dependent  SRCs. 

As  examination  of  Fig.  4.12  shows,  the  effects  of  the  elements  of 
eM  tend  to  swing  from  positive  to  negative  prior  to  approximately 
0.1  s.  This  effect  results  because  of  the  oscillatory  behavior  of  Q(t|a, 
eM)  that  can  be  seen  in  Fig.  4.9  and  derives  from  the  sine  and  cosine 
terms  in  the  closed-form  representation  for  Q(t|a,  eM)  shown  in 
Eq.  (3.31 ).  During  this  early  time  period,  all  variables  except  1  have 
appreciable  effects  on  Q(t|a,  eM).  After  approximately  0.1  s,  the 
oscillatory  behavior  of  Q(t)a,  eM)  has  significantly  decayed,  and  the 
uncertainty  in  Q(t|a,  eM)  is  dominated  by  C  and  E0.  As  indicated  by 
its  PCCs  and  SRCs,  1  has  a  strong  negative  partial  correlation  with 
Q(t|a,  eM)  beginning  at  about  0.1  s  but  the  actual  size  of  this  effect 
on  the  uncertainty  in  Q(t|a,  eM)  is  rather  small. 

In  this  example,  the  margin  Qm(t|a,  eM,  [tmn,  tmx])  is  not  an  affine 
transformation  of  the  underlying  analysis  result  Q(t|a,  eM).  As  a 
consequence,  sensitivity  analysis  results  for  Q(t|a,  eM)  cannot  be 
expected  to  be  the  same  as  sensitivity  analysis  results  for  Qm(t|a,  eM, 
[tmn,  tmx]).  For  this  reason,  a  sensitivity  analysis  for  Qm(t|a,  eM,  [tmn, 
tmxD  with  stepwise  regression  analysis  is  presented  in  Table  4.3.  This 
analysis  indicates  that  R  is  the  dominant  variable  affecting  the 
uncertainty  in  Qm(t|a,  eM,  [tmn,  tmx]),  with  Qm(t|a,  eM,  [tmn,  tTO]) 
tending  to  increase  as  R  increases.  After  R,  smaller  effects  on  Qm(t|a, 
eM,  [tmn,  tmx])  are  indicated  for  £0,  C  and  L,  with  Qm(t|a,  eM,  [tmn,  tmx]) 
tending  to  decrease  as  each  of  these  variables  increases. 

Additional  insights  on  the  effects  of  R,  E0,  C  and  L  on  Qm(t|a,  eM, 
[tmn,  tmx])  can  be  obtained  by  examining  the  scatterplots  involving 
these  variables  and  Qm(t|a,  eM,  [tmn,  tmx])  (Fig.  4.13).  Specifically, 
the  strong  positive  effect  of  R  on  Qm(t|a,  eM,  [tmn,  tmx])  can  be 
clearly  seen,  with  negative  values  for  Qm(t|a,  eM,  [tmn,  tmx]) 
occurring  for  small  values  for  R.  Further,  the  negative  but  less 
strong  effects  of  E0,  C  and  L  on  Qm(t|a,  eM}  [tmn,  tmx])  can  also  be 
seen,  with  negative  values  for  Qm(t|a,  eM,  [tmn,  tmx])  tending  to  be 
associated  with  large  values  for  E0,  C  and  L. 

Although  not  particularly  high,  the  final  R2  value  of  0.62  in 
Table  4.3  is  significantly  better  than  the  almost  meaningless  final 
R2  value  of  0.16  in  Table  4.2.  This  difference  results  because  the 
margins  associated  with  the  problem  in  Section  4.3  have  relation¬ 
ships  with  the  elements  of  eM  that  have  a  monotonic  character 
while  the  margins  in  Section  4.2  and  their  relationships  with  the 
elements  of  eM  do  not  have  this  character  (i.e.,  compare  the 
scatterplots  in  Figs.  4.8  and  4.13). 

4.4.  Epistemic  uncertainty  with  an  uncertain  bound 

The  QMU  results  presented  in  Sections  4. 1-4.3  involve  uniquely 
specified  bounds.  However,  it  is  likely  that  this  will  not  always  be  the 
case  in  QMU  analyses.  For  example,  a  requirement  might  be  that 

Table  4.3 

Stepwise  regression  analysis  to  identify  uncertain  variables  affecting  margin 
Qm(t|a,  eM,  [tm„,  t™,])  defined  in  Eq.  (4.40). 


Step3  Variable  SRC  R2 


1  R  0.67  0.45 

2  Eq  -0.26  0.53 

3  C  -0.23  0.58 

4  L  -0.20  0.62 


3  Table  structure  same  as 


in  Table  4.1. 
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Eq.  (4.40):  (a)  [R„  0„(t|a,  eMi,  [tmn,  Ul)l  i=U . nSE= 200,  (b)  [Eoi.  Qm(t|a,  eMi,  [tmn,  W])],  i=1,2 . 

(d)  [Lf,  (Utla,  e„i,  [tm„,  i=l,2 . nSE=200. 


a  certain  system  operates  but  the  conditions  that  define  when  the 
system  does  and  does  not  operate  appropriately  may  not  be  specified. 
Then,  it  is  the  responsibility  of  the  individuals  (i.e.,  analysts)  charged 
with  carrying  out  the  analysis  to  specify  the  conditions  under  which 
the  system  operates  in  the  manner  desired.  However,  there  may  be 
uncertainty  with  respect  to  exactly  what  conditions  are  necessary  for 
the  appropriate  operation  of  the  system.  Then,  in  this  situation,  there 
is  epistemic  uncertainty  as  to  the  conditions  that  should  be  specified 
to  define  what  constitutes  appropriate  operation  of  the  system. 

The  example  presented  in  Section  4.3  can  be  modified  to 
illustrate  this  situation.  As  originally  stated,  the  example  in 
Section  4.3  involves  a  bounding  interval  [Qj,.Q.l>]  for  Q(t|a,  eM) 
over  the  time  interval  [tmn,  tmx}-  For  the  example  of  this  section,  it 
is  assumed  that  the  specified  requirement  is  that  the  system  be 
operational  over  the  time  interval  [tmn,  tm*]  but  the  requirement 
does  not  specify  what  conditions  are  necessary  for  the  system  to 
be  operational.  For  purposes  of  illustration,  it  is  assumed  that  the 
analysts  involved  conclude  that  the  system  being  operational  over 
[tmn.  tmx]  corresponds  to  Q(t|  a,  eM)  being  within  a  bounding 
interval  [Qh,  Qh].  However,  they  are  uncertain  with  respect  to  the 
appropriate  value  for  this  bounding  interval.  Thus,  there  is 
epistemic  uncertainty  with  respect  to  the  values  to  use  for  Qb 


and  Qb.  As  a  result,  the  vector  eM  of  epistemically  uncertain  inputs 
to  the  analysis  now  has  the  form 

eM  =  [eK,eP]  =  [Qh,Qb,f,R,C,E0,;.],  (4.44) 

where  eR  =  [Q^.Qj,]  and  eP=  [L,  R,  C,  E0,  X]  as  indicated  in  Eq.  (3.21 ). 

For  purposes  of  illustration,  it  is  assumed  that  the  analysts 
conclude  that  (i)  Qb  is  contained  in  the  interval  [0.06, 0.08],  (ii)  Qb 
is  contained  in  the  interval  [0.14,  0.16],  (iii)  Qh  and  Qb  have  the 
same  uncertainty  structure  specified  for  L,  R0,  C,  E  and  X  (see 
Eqs.  (3.38)-(3.43)  and  associated  discussion),  and  (iv)  no  depen¬ 
dency  or  correlation  exists  between  Qh  and  Qb  (Fig.  4.14). 

This  problem  can  now  be  analyzed  exactly  as  in  Section  4.3. 
The  only  difference  is  that  eM  now  contains  7  rather  than 
5  elements,  with  two  of  these  elements  being  Qh  and  Qh. 
Specifically,  <5[Q(t|a,  eM):  tmn<  t<  tm*],  X+  and  X~  are  defined 
as  indicated  in  Eqs.  (4.35)-(4.37)  with  the  understanding  that  the 
indicator  function  <5[  —  ]  is  now  a  function  of  Q_b  and  Q_b-  Given 
the  dependency  of  <5[~]  on  and  Qbt  a  more  complete  but 
rather  cumbersome  notation  for  <5[~]  is  <5[~,(Qfi,  Qb)], 
which  will  be  used  below  to  make  the  dependence  of  §[  ~  ]  on 
Qh  and  Qb  explicit.  In  turn,  the  probabilities  of  compliance  and 
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Fig.  4.14.  Example  uncertain  bounding  interval  [Q^,  Q.6]  with  0.06  <  Q h  <  0.08 
and  0.14  <Qj,  <0.16  over  the  time  interval  [tmn,  tmJt]  =  [0.02,  0.18  s]  for  Q£t|a,  eM). 


Qm('|a.e„,[U.U) 


Fig.  4.15.  Estimated  CDF  summarizing  uncertainty  in  margin  Qm(t|a,  eM,  [tm„,  t„«]) 
defined  in  Eq.  (4.40)  for  time  interval  [tmn,  t„„]  =  [0.02,  0.18  s]  and  uncertain 
bounding  interval  [Q^,  Qh]  with  0.06  <  Qb  <  0.08  and  0.14  <  Q|,  <  0.1 6. 


noncompliance  are  given  by 


,  v+,  ^  <5[Q(t|a,eMi)  :tmn<t<  tmx,(Q_hi,(b,i)] 

Pem(*  )=  2^ - m - 


=  0.895 


(4.45) 


Pem(X-)  =  1-Pem(^+)  =  1-0.895  =  0.105,  (4.46) 

respectively,  where 

eM,  =  [Qw,Qw,l„Ri,C„E0i,;.1],  i=l,2 . nSE, 

is  now  an  LHS  of  size  nSE= 200  from  vectors  of  the  form  defined  in 
Eq.  (4.44). 

Margin  analysis  results  Qm(t|a,  eM,  [tmn,  tmx])  and  normalized 
margin  analysis  results  Qn(t|a,  eM,  [tmn,  tmx])  of  the  form  defined  in 
Eqs.  (4.40)  and  (4.41),  respectively,  can  also  be  obtained 
(Figs.  4.15  and  4.16). 

Similarly  to  the  results  in  Eqs.  (4.42)  and  (4.43),  “margin/uncer¬ 
tainty”  ratios  Qm/u(t|a,  eM,  [tm„,  tmx])  and  Qm/u(t  a,  eM,  [tmn,  t„K])  can  be 


Q„(f|a,  eM,[fm„,U) 


Fig.  4.16.  Estimated  CDF  summarizing  uncertainty  in  normalized  margin  Q„(t|a, 
eM,  [tm„,  t™,])  defined  in  Eq.  (4.41)  for  time  interval  [tmn,  tmx]=[0.02,  0.18  s]  and 
uncertain  bounding  interval  [Q^,  Qb]  with  0.06  S  s  0.08  and  0. 1 4  <  Qi,  <  0.1 6. 


Table  4.4 

Stepwise  regression  analysis  to  identify  uncertain  variables  affecting  margin 
Q„(t|a,  eM,  [tm„,  t™])  defined  in  Eq.  (4.40)  for  time  interval  [tm„,  tmx}  =  [0.02, 
0.18  s]  and  uncertain  bounding  interval  [Q^,  Qk]  with  0.06  <  <  0.08  and 

0.14  nQj,<  0.16. 


Step3  Variable  SRC 


R2 


R 

& 


0.53 


0.23 

0.13 


3  Table  structure  same  as  in  Table  4.1. 

used  to  summarize  the  distribution  for  Qm(t|a,  eM,  [tmn,  tmx])  in 
Fig.  4.15.  Specifically, 

0m/u(t|a,eM,[tmn,tmx]) 

f  0.01 1  /[0.01 1  -(-0.005)]  =  0.69  for  q  =  0.05 
=  |  0.011 /[0.011 -(-0.021)]=  0.34  for  q  =  0.00,  (447) 

and  similar  values  are  obtained  for  Qm/U  (t|a,  eM,  [tmn,  tmx])  as  a 
consequence  of  the  similarity  of  the  mean  and  median  values  for 
Q(t|a,  eM,  [tmn,  tmx])  (see  Fig.  4.15).  However,  as  is  always  the  case,  a 
significant  amount  of  information  is  lost  when  the  results  in  Figs.  4.14 
and  4.15  are  reduced  to  a  single  number  (see  Section  4.5  for 
additional  discussion). 

A  sensitivity  analysis  for  Qm(t|a,  eM,  [tmn,  tmx])  based  on 
stepwise  regression  analysis  is  presented  in  Table  4.4.  The  two 
most  important  variables  affecting  the  uncertainty  in  Qm(t|a,  eM, 
[tmn,  tmx])  are  R  and  Qb,  with  Qm(t|a,  eMf  [tmn.  tmx])  tending  to 
increase  as  R  increases  and  tending  to  decrease  as  increases.  In 
addition,  a  negative  effect  is  indicated  for  L  and  positive  effects 
are  indicated  for  Qb  and  £0. 

The  effects  of  R,  Qh  and  Qj,  on  Qm(t|a,  eM,  [tmn,  tmx])  can  be  seen  in 
the  scatterplots  in  Fig.  4.17.  In  particular,  negative  values  for  Qm(t|a, 
eM,  [tmn,  tmx])  tend  to  be  associated  with  small  values  for  R  that  occur 
in  conjunction  with  a  large  value  for  Qb  and  a  small  value  for  Qb. 

4.5.  Information  loss  in  a  “margin/uncertainty”  ratio 

As  already  emphasized  several  times,  results  of  the  form 
“margin/uncertainty”  involve  a  significant  loss  of  information.  This 
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Fig.  4.17.  Scatterplots  for  margin  Q„(t|a,  eM,  [tm „,  tmj )  defined  in  Eq.  (4.40)  for  time  interval  [tm„,  tmx]=[0.02,  0.18  s]  and  uncertain  bounding  interval  [Qk,  Chi]  with 

0.06  S  Qj,  <  0.08  and  0.14  <  Qn  s  0.16:  (a)  [R„  Q„(t|a,  em,  [tm„,  t™.])],  i=l,2 . nSE=200,  (b)  [Qj,,.,  0„(t|a,  eM,  [tm„,  t™,])],  i=l,2 . nSE=200,  and  (c)  [Qw,  Q^a,  eMi,  [tm„, 

WD],  1=1.2 . nSE= 200. 


loss  is  particularly  acute  because  the  actual  magnitudes  of  the 
margins  involved  in  the  determination  of  “margin/uncertainty”  are 
suppressed  and  cannot  be  determined  from  this  ratio.  Specifically, 
many  different  pairings  of  “margin”  and  “uncertainty”  can  result  in 
the  same  “margin/uncertainty”  ratio.  In  particular,  it  is  impossible 
to  determine  from  a  “margin/uncertainty”  ratio  whether  the 
underlying  margins  are  large  or  small.  Generally,  large  margins 
are  preferable  to  small  margins  but  insights  into  whether  the 
margins  underlying  a  “margin/uncertainty”  ratio  are  large  or  small 
are  not  directly  obtainable  from  this  ratio. 

The  ambiguity  of  a  “margin/uncertainty”  ratio  can  be  illu¬ 
strated  with  a  simple  plot  involving  the  ratio 
k=mb/(mb-ml),  (4.48) 

where  mb  is  the  best  estimate  for  a  margin  (e.g.,  the  mean  or 
median  of  the  epistemic  uncertainty  distribution  for  the  margin 
under  consideration),  m,  is  the  lower  estimate  for  a  margin 
(e.g.,  the  0.05  or  0.00  quantile  of  the  epistemic  uncertainty 
distribution  for  the  margin  under  consideration),  and  the  differ¬ 
ence  mb—mi  defines  the  “uncertainty”  in  the  margin  under 
consideration.  Then,  as  shown  in  Fig.  4.18,  infinitely  many  pairs 
( mb ,  mi)  of  margin  estimates  result  in  the  same  “margin/uncer¬ 
tainty”  ratio  k.  Specifically,  each  line  segment  in  Fig.  4.18  defines 
pairs  ( mb ,  mi)  of  margin  estimates  that  result  in  the  same 


Key:  k  =  mb  /  (mb  -  ni|)  =>  nri|  =  [(k- 1  )/k]  mb 


Fig.  4.18.  Relationship  of  best  estimate  margin  mb  and  lower  estimate  margin 
mi  to  “margin/uncertainty”  ratio  k  defined  by  k=mbj[mb  -mi). 
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“margin/uncertainty”  ratio  k.  As  a  consequence,  knowledge  of  the 
“margin/uncertainty”  ratio  k  provides  no  information  on  whether 
the  underlying  margins  involved  in  the  definition  of  k  are  large 
or  small. 

Some  additional  properties  of  the  “margin/uncertainty”  ratio  k  are 
also  illustrated  by  Fig.  4.18.  Specifically,  (i)  l<k<oo  results  for 
0<rrii<mt  with  k  approaching  oo  as  m,  approaches  mh  and  k 
approaching  1  as  m(  approaches  0,  (ii)  0  <  k  <  1  results  for  m(  <  0  <  mb 
with  k  approaching  1  as  mi  approaches  0  and  k  approaching  0  as  mt 
approaches  -  oo ,  and  (iii)  -  oo  <  k  <  0  results  for  mi<mb<  0  with  k 
approaching  0  as  mt  approaches  -  oo  and  k  approaching  -  oo  as  m, 
approaches  mb.  However,  as  already  discussed,  knowledge  of  k 
provides  no  information  on  the  underlying  margins  mb  and  mb 
Presumably,  the  half  plane  to  the  left  of  the  line  mb=m ,  in  Fig.  4.18 
is  not  of  interest  as  the  pairs  ( mb ,  m()  in  this  region  correspond  to  the 
best  estimate  margin  mb  being  less  than  the  lower  margin  estimate 
m,.  Also,  the  “margin/uncertainty”  ratio  k  is  not  defined  for  points  on 
the  straight  line  mb=mi  as  this  situation  involves  an  undefined 
division  by  mb-mt=  0. 

Significant  reservations  about  the  use  of  “uncertainty/margin” 
ratios  are  also  expressed  in  the  NAS/NRC  report  on  QMU  (e.g., 
Finding  1-4,  p.  25,  Ref.  [77]). 


5.  QMU  with  aleatory  and  epistemic  uncertainty: 
characterization  with  probability 

The  use  of  probability  to  represent  epistemic  uncertainty  in 
analyses  that  involve  only  epistemic  uncertainty  is  discussed  and 
illustrated  in  Sections  3.3,  3.4  and  4.  Specifically,  the  formal 
discussion  in  Section  3.3  involves  a  generic  real-valued  quantity 
y(t|a,eM)=/(t|a,eM)  (5.1) 

conditional  on  a  specific  realization  a  of  aleatory  uncertainty.  The 
vector  eM  contains  epistemically  uncertain  analysis  inputs,  with 
the  uncertainty  in  these  inputs  characterized  by  a  probability 
space  (£M,  EM,  pEM). 

As  discussed  in  Section  3.5,  an  increase  in  complexity  is  to 
include  the  aleatory  uncertainty  associated  with  a  in  the  analysis. 
Then,  in  addition  to  the  probability  space  (EM,  EM,  pEM)  that 
characterizes  the  epistemic  uncertainty  associated  with  eM,  there 
is  also  a  probability  space  (A,  A,  pA)  that  characterizes  the 
aleatory  uncertainty  associated  with  a.  Further,  there  can  be, 
and  often  is,  epistemic  uncertainty  with  respect  to  a  vector  e„  of 
quantities  used  in  the  definition  of  the  probability  space  ( A ,  A, 
pA).  As  a  result,  there  is  also  a  probability  space  (£A,  EA,  pM)  that 
characterizes  the  epistemic  uncertainty  associated  with  eA.  The 
vector  e=[e>i,  eM]  then  contains  the  epistemically  uncertain 
inputs  to  the  analysis,  with  the  uncertainty  in  e  characterized 
by  a  probability  space  {£,  E,  pE)  that  derives  from  the  probability 
spaces  (EA,  EA,  pM)  and  (EM,  EM,  pEM).  Conceptually,  the 
resultant  analysis  involves  the  three  basic  analysis  components 
discussed  in  Section  3.2:  (i)  EN1,  a  probabilistic  characterization 
of  aleatory  uncertainty  (i.e.,  a  probability  space  (A,  A,  pA) 
that  characterizes  the  aleatory  uncertainty  associated  with  the 
elements  of  a),  (ii)  EN2,  a  model  that  predicts  system  behavior 
(i.e.,  a  function /[t | a,  eM)),  and  (iii)  EN3,  a  probabilistic  character¬ 
ization  of  epistemic  uncertainty  (i.e.,  a  probability  space 
(£,  E,  pi:)  that  characterizes  the  epistemic  uncertainty  associated 
with  the  elements  of  e= [e^,  eM]). 

The  results  of  analyses  involving  aleatory  and  epistemic 
uncertainty  are  usually  summarized  with  CDFs  and  CCDFs  that 
display  the  effects  of  aleatory  uncertainty  conditional  on  specific 
realizations  of  epistemic  uncertainty  and  also  with  various 
quantities  derived  from  such  CDFs  and  CCDFs  (e.g.,  quantiles 
and  expected  values).  In  turn,  margins  can  be  defined  in  a  variety 


of  ways  for  CDFs,  CCDFs  and  associated  derived  quantities,  and 
the  presence  of  epistemic  uncertainty  results  in  a  corresponding 
epistemic  uncertainty  in  the  resulting  margins. 

This  section  uses  the  function  A(t|  a,  eM)  introduced  in  Section  3.6 
to  illustrate  two  ways  in  which  QMU  analyses  could  arise  and  be 
carried  out  in  the  context  of  analyses  that  involve  a  generic  result 
y(t|a,  eM)  of  the  form  indicated  in  Eqs.  (3.24)  and  (5.1).  Further, 

e=  [eA,eM]  =  [ei,e2,e3,e4,e5]  =  [/.,a,m,b,r],  (5.2) 

where  (i)  eA=[k,  a,  m,  b]  and  eM=[r]  have  the  properties  defined  in 
conjunction  with  Eq.  (3.59)  and  (ii)  the  corresponding  probability 
space  (£,  E,  pE)  that  characterizes  the  epistemic  uncertainty  asso¬ 
ciated  with  e  is  defined  in  conjunction  with  Eqs.  (3.60)-(3.65). 

The  time-dependent  behavior  of  A(t|a,  eM)=A(t|a,  r)  is  illustrated 
in  Figs.  3.9  and  3.10,  and  the  CDFs  and  CCDFs  for  A(10|a,  eM)= 
A(10|a,  r)  that  result  for  different  values  of  e  are  illustrated  in 
Figs.  3.11  and  3.12  and  are  defined  by  the  probabilities  pA[A(~\0 
a,  r)<A  e^]  and  pA[A <A(10|a,  r)|eA],  respectively.  As  indicated  by 
the  vertical  line  “|",  the  value  ofA(t|a,  r)  is  conditional  on  a  and  r.  As  a 
result,  e„  does  not  affect  the  value  of  A(t|a,  r)  but  does  affect  the 
distribution  of  A(t|a,  r)  arising  from  the  distribution  of  possible  values 
for  a.  In  contrast,  probabilities  of  the  form  pA[A(t  a,  r)<A  e^]  and 
pA[A  <  A(t|a,  r)|e/i]  are  conditional  on  eA  and  hence  on  the  probability 
space  (A,  A,  pA )  with  associated  density  function  dA  (a|e,i). 

The  examples  presented  in  this  section  use  an  LHS 

?i  =  [C/ii.e/wi] 

=  [en,ei2 . ei5] 

=  (4a(,mi,bi,§|,  i=l,2 . nSF  =  200,  (5.3) 

from  E  generated  in  consistency  with  the  distributions  that  define 
the  probability  space  (£,  E,  pE).  Further,  results  conditional  on 
individual  sample  elements  e,  are  generated  with  a  random 
sample 

a j,  j  =  1, 2, ...,nSA=  10, 000,  (5.4) 

from  A  consistent  with  the  probability  space  (A,  A,  pA).  As  a  result 
of  the  values  associated  with  eA=[2j,  ait  mb  fa,],  the  sample  space 
(A,  A,  pA)  underlying  the  generation  of  the  sample  in  Eq.  (5.4) 
changes  for  each  sample  element  ej=[e,4„  eMi]  =  [e^,-,  r,].  Evalua¬ 
tion  of  A(t|aj,  r.)  and  results  such  as  pA[A  <  A(Ca,  r,)|eA]  for 
elements  of  the  preceding  samples  generates  mappings  of  the 
form 

[rI-,>V(t|aj,rI)],  i  =  1,2 . nSE  =  200,  j=  1,2,. .  „nSA=  10,000 

(5.5) 

and 

{ei,pA[A<A(t|a,ri)|eyti]},  i=l,2 . nSE=  200,  (5.6) 

that  are  used  in  the  generation  of  the  example  results  presented 
in  this  section. 

The  following  topics  related  to  QMU  in  the  presence  of 
aleatory  and  epistemic  uncertainty  are  considered  in  this  section: 
epistemic  uncertainty  in  margins  associated  with  a  specified  bound 
on  a  quantile  deriving  from  aleatory  uncertainty  (Section  5.1),  and 
epistemic  uncertainty  in  margins  associated  with  a  specified 
bound  on  an  expected  value  deriving  from  aleatory  uncertainty 
(Section  5.2). 

As  indicated  at  the  beginning  of  Section  3.5,  the  NAS/NRC 
report  on  QMU  recommends  the  use  of  what  it  describes  as  the 
“probability  of  frequency  approach”  in  QMU  analyses  (Recom¬ 
mendation  1-7,  p.  33,  and  App.  A,  Ref.  [77]).  The  examples 
presented  in  Sections  5.1  and  5.2  involve  what  the  NAS/NRC 
report  describes  as  the  “probability  of  frequency  approach”  (i.e., 
an  analysis  that  involves  an  explicit  separation  of  aleatory  and 
epistemic  uncertainty). 
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5.1.  Epistemic  uncertainty  with  a  specified  bound  on  a  quantile 

For  this  example,  it  is  assumed  that  p^[20<A(10|a,  r) | ]  is 
required  to  be  less  than  a  bound  (e.g.,  the  possible  bounds 
Pm  =  0.05  and  pb2  =  0.1  in  Fig.  5.1b).  Specifically,  the  values  for 
p,t[20<A(10|a,  r) | eA]  in  Fig.  5.1b  correspond  to  the  exceedance 
probabilities  associated  with  the  vertical  line  in  Fig.  5.1a,  and  the 
corresponding  distribution  of  these  probabilities  and  the  asso¬ 
ciated  bounds  Pm  and  pb2  are  shown  in  Fig.  5.1b.  In  particular,  the 
probabilities  that  p„[20<  A(1 0|a,  r)|eA]  will  exceed  pbl  =0.05  and 
pb2=0.1  are  0.055  and  0.025,  respectively.  The  indicated  excee¬ 
dance  probabilities  of  0.055  and  0.025  derive  from  epistemic 
uncertainty  and  thus  characterize  degrees  of  belief  that 
Pzi[20  <  A(10|a,  r)|eA]  will  exceed  pbl  and  pb2,  respectively. 

In  turn,  the  margins  between  pA[20  <  A(10|a,  r)|eA]  and  the 
bounds  pbk,  k=  1,  2,  indicated  in  Fig.  5.1b  can  be  defined  in  the 
same  manner  as  the  margins  in  Eq.  (4.5).  Specifically,  the  margin 
pmfc(10|e)  is  defined  by 

Pmk(10|e)  =  pbk-p2i[20  <A(10|a,r)|e2i],  (5.7) 

with  pmJ((10|e)  >  0  indicating  that  bound  pbk  is  satisfied  and 
Pmk(l  0 1  e)  <  0  indicating  that  bound  pbk  is  not  satisfied.  As  a  result 


of pA [20 <A(10| a,  r)|e„]  being  epistemically  uncertain,  the  corre¬ 
sponding  margins  pmb(10  e)  are  also  epistemically  uncertain  and 
have  an  uncertainty  structure  that  derives  from  the  correspond¬ 
ing  uncertainty  structure  assumed  for  e  (Fig.  5.2). 

As  discussed  in  conjunction  with  Eq.  (4.6),  an  alternative 
presentation  involves  the  use  of  normalized  margins.  For  the 
present  example,  normalized  margins  are  defined  by 

P„k(l  0 1  e)  =  pmk(l  0 1  e)/pbk  (5.8) 

for  k=l,  2  and  express  margin  as  a  fraction  of  the  corresponding 
bounding  value  (Fig.  5.3). 

If  desired,  the  CDFs  for  margin  in  Fig.  5.2  can  be  converted  into 
summary  “margin/uncertainty”  results  as  indicated  in  Eqs.  (4.7) 
and  (4.8)  by  the  normalizations 

Pm/«(10)  =  pmio.5(10)/[pm,o.5(10)— pm,(10)]  (5.9) 

Pm/U(10)  =pm(10)/[pm(10)-pmq(10)],  (5.10) 

where  pmq(  10)  is  the  q  quantile  (e.g.,  q  =  0.0,  0.05  or  0.5)  for  the 
margin  pm(10|e)  corresponding  to  pmi(10|e)  in  Fig.  5.2a  or 
Pm2(10|e)  in  Fig.  5.2b  and  pm(10)  is  the  expected  value  for 


a  b 


a  b 


Fig.  5.2.  Estir 


CDFs  for  margins  pmfc(10|e)  associated  with  bounds  pbk  for  lc=l,  2:  (a)  pml(10|e)  for  pM=0.05,  and  (b)  pm2(  1 0 1 e)  for ; 
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pm(10|e).  In  turn, 

J  0.050/[0.050-(-0.007)]  =  0.88  for  q  =  0.05 
Pm/u, i(10)  =  |  o.050/[0.050-(-0.313)]  =  0.14  for  q  =  0.00 

(5.11) 


f  0.041 /[0.041 -(-0.007)]  =  0.85  forq  =  0.05 
Pm/u,iO0)=  j0.041/[0.041 -(-0.313)]  =  0.12  for  q  =  0.00 

(5.12) 

for  pml(10|e)  in  Fig.  5.2a,  and 

J  0.100/(0.100-0.043)  =  1.75  for  q  =  0.05 
Pm/u, 2(1°)  =  |  o,100/[0.100-(-0.263)]  =  0.28  for  q  =  0.00 

(5.13) 


(0.091/(0.091-0.043)=  1.90  for  q  =  0.05 
Pm/u. 2(10)  =  <  Q  ogl  /[0.091 -(-0.263)]  =  0.26  for  q  =  0.00  (514) 

for  pmi(10|e)  in  Fig.  5.2b.  The  normalizations  in  Eqs.  (5.11)  and 
(5.12)  are  the  outcomes  of  converting  all  the  information  in 
Figs.  5.1  and  5.2a  into  single  numbers.  Similarly,  the  normal¬ 
izations  in  Eqs.  (5.13)  and  (5.14)  are  the  outcomes  of  converting 
all  the  information  in  Figs.  5.1  and  5.2b  into  single  numbers. 
Because  of  the  presence  of  both  aleatory  and  epistemic  uncer¬ 
tainty,  the  conversion  of  analysis  results  into  a  single  “margin/ 
uncertainty”  ratio  as  illustrated  in  this  section  involves  a  greater 
loss  of  information  than  is  the  case  when  only  epistemic  uncer¬ 
tainty  is  present  (see  Section  4.5  for  additional  discussion). 

Additional  insights  with  respect  to  the  uncertainty  associated  with 
the  margins  pml(10|e)  and  pm2(10|e)  in  Fig.  5.2  can  be  obtained  by 
performing  a  sensitivity  analysis  on  the  values  for  pA[20  <  A(10|a, 
r)|e/i]  summarized  in  Fig.  5.1  and  used  in  the  generation  ofpml(10je) 


Table  5.1 

Stepwise  regression  analysis  to  identify  uncertain  variables  affecting  exceedance 
probability  pA[ 20  <a(10|a,  r)|eA]. 

Step3  Variable  SRC  it2 


1  r  -0.36  0.14 

2  X  0.24  0.19 


3  Table  structure  same  as  in  Table  4.1. 


and  pm2(10|e)  as  indicated  in  Eq.  (5.7).  Because  pmi(10|e)  and 
pm2(10|e)  are  obtained  from  an  affine  transformation  of 
p„[20<A(10|a,  r)|eA],  the  analysis  of  pA[20  < A(10|a,  r)|eA]  produces 
effectively  the  same  results  as  an  analysis  of  pml(10|e)  and  pm2(10|e). 
The  only  difference  is  that  the  effects  of  individual  variables  are 
reversed  owing  to  the  subtraction  of  p„[20<A(10|a,  r)|eA]  in  the 
definition  of  pmi(10|e)  and  pm2(10]e)  in  Eq.  (5.7). 

An  initial  sensitivity  analysis  for  pA[20  <  A(10|a,  r)  eA]  based  on 
stepwise  regression  analysis  is  presented  in  Table  5.1.  This  analysis 
is  basically  a  failure  as  it  produces  a  regression  model  containing 
the  variables  r  and  X  that  has  an  R2  value  of  only  0.19.  As  a  result, 
this  regression  model  provides  little  information  on  the  variables 
that  are  affecting  the  uncertainty  in  pA[20  <  A(10|a,  r)|e/i]  Fig.  5.3. 

The  natural  next  step  at  this  point  is  to  examine  scatterplots 
involving  pA[20  < A(10|a,  r)  eA]  and  the  elements  of  e  (Fig.  5.4).  A 
clearer  picture  of  the  effects  of  r  and  X  on  pA[20  <A(10|a,  r)  e^] 
emerges  from  an  examination  of  these  plots.  Specifically, 
Pa[20  <A(10|a,  r)|e/i]  decreases  as  r  increases  and  is  almost 
always  zero  when  r  exceeds  approximately  0.75.  Further,  zero 
values  for  pA[20  <A(10|a,  r)|e„]  show  a  strong  tendency  to  be 
associated  with  values  for  X  that  are  less  than  approximately  1.0. 

The  failure  of  the  regression  analysis  in  Table  5.1  results 
because  the  large  number  of  zero  values  for  pA[20  <  A(10  a,  r)  eA] 
results  in  patterns  that  the  linear  regression  model  in  use  cannot 
match.  In  such  situations,  there  are  a  number  of  additional  techni¬ 
ques  for  sampling-based  sensitivity  analysis  that  can  be  tried.  The 
examination  of  scatterplots  as  illustrated  is  certainly  the  simplest  of 
these  techniques.  Other  possibilities  include  rank  regression,  tests 
for  patterns  based  on  gridding,  nonparametric  regression,  tests  for 
patterns  based  on  distance  measures,  tree-based  searches,  the  two- 
dimensional  Kolmogorov-Smimov  test,  and  the  squared  differences 
of  ranks  test  [53,54,56], 


5.2.  Epistemic  uncertainly  with  a  specified  bound  on  an  expected  value 

For  this  example,  it  is  assumed  that  the  expected  value  E/i[A(10|a, 
r)|eyi]  summarized  in  Fig.  3.13  is  required  to  be  less  than  a  bound  (e.g„ 
the  bound  Ab  =  13  in  Fig.  5.5).  At  a  conceptual  level,  this  example  is 
essentially  the  same  as  the  example  in  Section  5.1  as  the  only 
difference  is  that  (i)  each  CCDF  in  Section  5.1  is  being  reduced  to 
an  exceedance  probability  pA[20  <  A(10|a,  r)  eA]  associated  with 
A(10|a,  r)=20  and  (ii)  each  CCDF  in  the  present  section  is  being 
reduced  to  an  expected  value  EA[A(10|a,  r) | e^].  In  both  cases,  CCDFs 
summarizing  aleatory  uncertainty  are  being  reduced  to  a  single 


a 


b 


Fig.  5.3.  Estir 


CDFs  for  normalized  margins  pnfc(10|e)  associated 


;  pbk  for  k=  1,  2:  (a)  pnl(10|e)  for  pM=0.05,  and  (b)  pn2(10|e) 


2  =  0.1. 
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a  b 


Fig.  5.4.  Scatterplots  for  exceedance  probability  p„[20  <A(10  a,  r)|e^]:  (a)  (r„  p/1[20<J1(10|a,  r,-)|eAi]5,  i=l,2 . nSE= 200,  and  (b)  (2*  pA[20<A(10|a,  r,)|ea(]),  1=1,2 . 

nSF= 200. 


number.  However,  it  is  easy  to  envision  that  each  of  these  cases  could 
arise  in  QMU  analyses.  Specifically,  the  results  in  Section  5.1  involve  a 
situation  in  which  a  bound  is  being  placed  on  the  likelihood  of 
extreme  outcomes  arising  from  aleatory  uncertainty,  and  the  results 
in  the  present  section  involve  a  situation  in  which  a  bound  is  being 
placed  on  the  expected  value  of  outcomes  arising  from  aleatory 
uncertainty. 

Margins  and  normalized  margins  for  Ea[A(  10 1  a,  r)  |  e^]  are 


defined  by 

Am(l  0 1  e)  =  A„  -£„[A(  10 1  a,r)  |  eA] 

(5.15) 

and 

Ai(10  e)  =Am(\0\e)/Ab, 

(5.16) 

respectively,  and  summarized  in  Fig.  5.6. 

Similarly  to  the  results  in  Eqs.  (5.9)  and  (5.10),  the  CDF  for 
margin  in  Fig.  5.6a  can  be  converted  into  summary  “margin/ 
uncertainty”  results  by  the  normalizations 


Am/uO  0)  =  A„, 0.5(1 0)/[Am.o.s(l  0)-Am,(l  0)] 

8.7/(8.7— 0.6)=  1.1  for  q  =  0.05 
87/8.7— [(— 5.3)]  =  0.6  for  q=  0.00 


and 

A„/U(l  0)  =  Arc(l  0)/[Im(l 0)-Amj(l  0)] 

7. 8/(7. 8-0.6)  =  1.1  for  q  =  0.05 
7.8/[7.8 — ( — 5.3)]  =  0.6  for  q  =  0.00,  (518) 

where  Anq(10)  is  the  q  quantile  for  the  margin  Am(10|e)  and 
An(10)  is  the  expected  value  for  Am(10  e).  The  preceding  normal¬ 
izations  are  the  outcomes  of  converting  all  the  information  in 
Figs.  3.11,  3.12,  5.5  and  5.6a  into  single  numbers  (see  Sections  4.5 
for  additional  discussion). 

A  sensitivity  analysis  for  EA[A(10|a,  r) | eA]  based  on  stepwise 
regression  analysis  is  presented  in  Table  5.2.  As  indicated,  the 
uncertainty  in  EA[A(10|a,  r) | ]  is  dominated  by  r  and  A,  with 
smaller  effects  indicated  for  m  and  a.  Specifically,  EA[A(10|a,  r)  eA] 
tends  to  decrease  as  r  increases  and  tends  to  increase  as  each  of  A, 
m  and  a  increases.  The  final  regression  model  has  an  R2  value  of 
0.85,  which  indicates  that  most  of  the  uncertainty  in  £a[A(10  a, 
r)|eA]  is  being  captured  by  the  regression  model.  A  regression- 
based  sensitivity  analysis  for  Am(10|e)  would  produce  the  same 
results  as  shown  in  Table  5.2  with  the  exception  that  the  signs  on 
the  SRCs  would  be  reversed  as  a  result  of  the  subtraction  of 
EA[A(10|a,  r) | e^]  in  the  definition  of  Am(10|e)  in  Eq.  (5.15). 

For  perspective,  the  scatterplots  for  the  two  dominant  vari¬ 
ables  affecting  the  uncertainty  in  £„[A(  10  a,  r)  eA]  identified  in  the 
regression  analysis  in  Table  5.2  are  shown  in  Fig.  5.7.  Specifically, 
the  negative  effect  of  r  and  the  positive  effect  of  A  are  easily  seen 
in  the  two  scatterplots  in  Fig.  5.7. 


6.  Summary  discussion 

As  indicated  by  the  name,  QMU  involves  three  concepts: 
quantification,  margin  and  uncertainty.  These  concepts  are  dis¬ 
cussed  in  the  following  sections:  Margins  (Section  6.1),  uncer¬ 
tainty  (Section  6.2),  and  quantification  (Section  6.3).  Then,  the 
presentation  of  QMU  results  is  discussed  (Section  6.4). 

6.1.  Margins  in  QMU 

Intuitively,  a  margin  M  is  a  measure  of  the  difference  between 
a  requirement  R  placed  on  the  performance  of  a  system  and  the 
predicted  performance  P  of  the  system,  with  M  >  0  indicating  that 
the  requirement  is  met  and  M  <  0  indicating  that  the  requirement 
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Table  5.2 

Stepwise  regression  analysis  to  identify  uncertain  variables  affecting  tyAflO 
|a,  r)|e*]. 


Step3  Variable  SRC  if2 


3  Table  structure  same  as  in  Table  4.1. 


is  not  met.  More  explicitly,  M  is  a  function  M(R,  P)  of  R  and  P  with 
the  properties  that 

M(R,P)  >  0  =>■  compliance  of  performance  P  with  requirement  R 

(6.1) 

M(R,P )  <  0  =>•  noncompliance  of  performance  P  with  requirement  R. 

(6.2) 

If  R  and  P  are  single  numerical  values,  then  the  definition  of  M(R,  P) 
could  be  as  simple  as 

M(R,P)  =  R—P  (6.3) 

if  P  is  required  to  be  less  than  or  equal  to  R  and 

M(R,P)  =  P—R  (6.4) 

if  P  is  required  to  be  greater  than  or  equal  to  R.  Margins  of  this 
type  are  extensively  discussed  and  illustrated  in  Sections  4  and  5 
and  also  in  Refs.  [79,80], 

In  many  real  analyses,  it  is  unlikely  that  R  and  P  will  be  single 
numbers.  Rather,  greater  complexity  in  the  definition  of  R  and  P  is 
likely  to  be  the  case.  For  example,  R  might  correspond  to 
requirements  on  several  subsystems  of  a  larger  system,  and  P, 
in  turn,  would  correspond  to  the  performance  of  these  subsys¬ 
tems.  As  another  example,  R  and  P  might  be  functions  or  vectors 
of  functions.  For  this  reason,  the  requirement  R  and  performance 
P  are  more  appropriately  represented  as  the  vectors  R  and  P.  The 
analyses  presented  in  Ref.  [79]  are  of  this  type  with  multiple 
requirements  placed  on  system  performance.  Then,  M  is 


b 


a  function  M(R,  P)  with  the  properties  that 

M(R,P)  >  0  =>  compliance  of  performance  P  with  requirement  R 

(6.5) 

and 

M(R,P)  <  0  =>  noncompliance  of  performance  P  with  requirement  R. 

(6.6) 

In  this  situation,  the  evaluation  of  M(R,  P)  involves  a  more 
complex  calculation  than  the  simple  subtractions  indicated  in 
Eqs.  (6.3)  and  (6.4). 

In  practice,  it  is  also  possible  for  M(R,  P)  itself  to  be  a  vector 
and  thus  appropriately  denoted  by  M(R,  P).  For  example,  if 
R=[Ri,R2,...,R„r],  (6.7) 

P=[Pi.P2 . P„K]  (6.8) 

and 

Mr(Rr,Pr)  =  Rr—Pr  (6.9) 

is  the  margin  associated  with  requirement  Rr  for  r=l,2 . nR, 

then  M(R,  P)  might  be  defined  by 

M(R,P)  =  [M1(R1,P1),M2(R2,P2)„  .  .,MnR(RnR,PnR)] 

=  [R1-P1,R2-P2 . RnR-P„R\.  (6.10) 

In  a  situation  of  this  type,  one  possibility  is  to  (i)  consider  each  of 
the  margins  Mr(Rr,  Pr)  separately  and  (ii)  define  the  corresponding 
real-valued  margin  M(R,  P)  to  be  the  minimum  of  the  values  for 
the  individual  margins  M,(Rr,  Pr).  This  is  the  situation  illustrated 
in  Sections  4.2-4.4  and  in  Ref.  [80]  and  also  appears  to  be  the 
intent  of  the  regulatory  requirements  and  associated  margins 
illustrated  in  Ref.  [79],  Another  possibility  is  the  use  of  some  type 
of  weighted  average  to  reduce  the  margins  M,(Rr,  Pr )  to  a  single 
number.  In  general,  the  reduction  of  M(R,  P)  to  a  single  real¬ 
valued  margin 

M(R,P)  =/[M(R,P)]  (6.11) 

through  the  application  of  a  suitable  function  /  is  likely  to  be 
analysis  specific  and  outside  the  scope  of  this  discussion. 

6.2.  Uncertainty  in  QMU 

If  R  and  P  were  known  with  complete  certainty  and  the 
function  M(R,  P)  that  defined  the  margin  associated  with  R  and 
P  was  unambiguously  defined,  then  there  is  no  uncertainty  and 
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Fig.  5.7.  Scatterplots  for  E„[71(10|a,  r)|e„]:  (a)  (r„  EyiWlOja,  r.OM),  i=  1,2 . nSE= 200,  and  (b)  (1,,  £,,[71(10  a. 


),  1  =  1,2 . nSE= 200. 


the  margin  M(R,  P)  is  uniquely  defined.  Unfortunately,  this  is 
unlikely  to  be  the  case  in  a  real  analysis.  In  the  analyses  of  most 
systems,  there  is  likely  to  be  significant  uncertainties  associated 
with  the  determination  of  P,  and  uncertainty  with  respect  to  the 
appropriate  definition  of  R  is  also  possible.  However,  this  discus¬ 
sion  assumes  there  is  no  uncertainty  in  the  definition  of  the 
function  M(R,  P)  that  converts  R  and  P  into  a  margin  M.  The 
examples  in  Ref.  [79]  illustrate  analyses  of  real  systems  that 
involve  significant  uncertainties  in  the  modeling  of  system 
performance  P.  Further,  the  notional  analyses  in  Section  4.4  and 
in  Section  3.6  of  Ref.  [80]  involve  uncertainty  in  both  R  and  P.  The 
possibility  of  uncertainty  in  both  R  and  P  is  recognized  by  the 
NNSA  in  Quote  (NNSA-3). 

The  presence  of  uncertainty  in  the  determination  of  M(R,  P) 
can  be  acknowledged  through  the  introduction  of  a  vector 

eM  =  [eMi  ,Om . eM,nEM]  (6.1 2) 

of  uncertain  analysis  inputs  required  in  the  evaluation  of  1W(R,  P). 
With  this  introduction,  M(R,  P)  is  appropriately  represented  by 
M(R,P|eM)  =  margin  determined  with  value  for  P  and  possibly 
the  value  for  R  conditional  on  the  values 


for  eMi,eM2 . eM,„£  contained  in  eM. 

(6.13) 

More  specifically, 

M(R,P\eM)  =  M[R,P(eM)] 

(6.14) 

if  only  P  depends  on  eM; 

M(R,P|eM)  =  M[R(eM),P(eM)] 

(6.15) 

if  R  and  P  depend  on  eM;  and 

M(R,P|eM)  =  M[R(eR),P(eP)],  eM  =  [eR,eP], 

(6.16) 

if  eM  can  be  decomposed  into  a  vector  eR  that  contains  only  variables 
affecting  R  and  a  vector  eP  that  contains  only  variables  affecting 
P.  Section  4.4  and  also  Section  3.6  of  Ref.  [80]  illustrate  analyses  in 
which  eM  can  be  decomposed  into  a  vector  eR  and  a  vector 
eP;  specifically,  eR  and  eP  correspond  to  [Q^.Q/dand  [L,R,C,£o,2], 
respectively,  in  these  sections. 

The  values  for  the  variables  contained  in  eM  are  assumed  to  be 
uncertain  in  the  sense  that  the  analysis  leading  to  M(R,  P)  has 
been  designed  on  the  assumption  that  the  appropriate  value  for 
M(R,  P)  will  be  obtained  if  the  appropriate  values  for  the  elements 
of  eM  are  used.  Unfortunately,  the  appropriate  value  to  use  for  eM 


is  not,  and  cannot  be  in  most  analyses,  known  with  certainty. 
Rather,  there  is  uncertainty  with  respect  to  the  appropriate  value 
to  use  for  each  element  of  eM.  Uncertainty  with  respect  to  the 
value  of  a  fixed,  but  poorly  known,  quantity  is  usually  referred  to 
as  epistemic  uncertainty,  which  is  why  eM  is  used  as  the 
designator  for  the  vector  of  uncertain  quantities  in  Eq.  (6.12). 
Alternative  descriptors  used  for  epistemic  uncertainty  include 
subjective  uncertainty,  state  of  knowledge  uncertainty,  and  redu¬ 
cible  uncertainty  (see  Section  2  for  additional  discussion). 

The  epistemic  uncertainty  associated  with  the  elements  of  eM 
is  usually  characterized  with  probability;  however,  alternative 
mathematical  structures  for  the  characterization  of  epistemic 
uncertainty  such  as  possibility  theory  and  evidence  theory  also 
exist  [80,117],  With  the  use  of  probability  to  characterize  epis¬ 
temic  uncertainty,  the  uncertainty  associated  with  eM  is  defined 
by  a  sequence  of  distributions 

Di,D2 . DnEM,  (6.17) 

where  the  distribution  Dr  associated  with  element  eMr  of  eM 
provides  a  mathematical  characterization  of  the  available  infor¬ 
mation  with  respect  to  where  the  appropriate  value  for  eMr  is 
located  for  use  in  evaluation  of  M(R,  P).  Correlations  and  other 
restrictions  involving  relationships  between  individual  elements 
of  eM  may  also  be  present.  Distributions  of  the  form  indicated  in 
Eq.  (6.17)  are  often  developed,  at  least  in  part,  through  an  expert 
review  process  [100-106],  In  turn,  the  distribution  for  eM  that 
results  from  the  distributions  in  Eq.  (6.17)  and  any  associated 
restrictions  leads  to  a  resultant  distribution  of  values  for  M(R,  P| 
eM)  that  characterizes  the  (epistemic)  uncertainty  with  respect  to 
the  appropriate  value  for  the  margin  M(R,  P). 

The  distributions  Di,  D2 . DnEM  indicated  in  Eq.  (6.17)  are,  in 

essence,  defining  a  probability  space  (£M,  EM,  pEM)  for  epistemic 
uncertainty,  where  £M  is  the  set  of  possible  values  for  eM,  EM  is  a 
suitably  restricted  set  of  subsets  of  EM,  and  pEM  defines  the 
probabilities  for  individual  sets  contained  in  EM.  In  practice,  the 
individual  distributions  Dr  in  Eq.  (6.17)  are  usually  defined  by  CDFs 
or  CCDFs  that,  in  effect,  define  EM,  EM,  pEM  and  an  associated 
density  function  dEM(eM)  defined  on  EM.  Although  EM,  EM,  pEM 
and  dEM(eM)  are  notationally  useful  and  can  be  formally  defined, 
actual  calculations  involving  the  probability  space  (£M,  EM,  pEM) 

are  usually  carried  out  with  direct  use  of  the  distributions  Dj,  D2 . 

DnEM  and  any  associated  restrictions  involving  these  distributions 
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(e.g.,  by  using  random  or  Latin  hypercube  sampling  to  generate 
values  for  eM  consistent  with  the  distributions  D,,  D2 . DnrM). 

Once  the  distributions  Dj ,  D2,  ....  DnEM  in  Eq.  (6.1 7)  are  defined, 
and  hence  the  probability  space  ( EM ,  EMI,  pEM)  is  also  defined,  the 
uncertainty  in  the  margin  M(R,  P|eM)  can  be  formally  represented 
by  a  CDF  or  a  CCDF.  Specifically,  the  CDF  and  CCDF  for  M(R,  P|eM) 
are  formally  defined  by 

p£M[M(R,P|eM)  <M]  =  j^<yM(R,P|eM)]dEM(eM)d£M  (6.18) 
and 


pEM[M  <  M( R,P|eM)]  =  J^dM[M(R,P\eM)]dEM(eM)dEM,  (6.19) 


respectively,  where  pEM  denotes  epistemic  probability,  d£M 
represents  an  increment  of  volume  from  EM,  and  the  indicator 
functions  <5Jvf(~)  and  dM(~)  are  defined  by 


aM[M(R,P|eM)]  = 


1  ifM(R,P|eM)<M 
0  otherwise 


and 


<5M[M(R,P|eM)]  = 


1  ifM(R,P|eM)>M 
0  otherwise. 


In  practice,  the  indicated  probabilities  are  usually  approximated 
by 


pm[M(R,P  eM)  <  M]  =  £  <iM[M(R,P| eMi)]/nS  (6.20) 


pEM[M<  JW(R,P  |  eM)]  m  £  (jM[M(R,P|eMi)]/nS,  (6.21) 

where  eM;,  f  =  1 ,2,  ...,  nS,  is  a  random  or  Latin  hypercube  sample  of 
size  nS  generated  from  EM  in  consistency  with  the  distributions 

D-i,  D2 . DnEM  and  any  associated  restrictions.  Calculations  of 

this  type  are  discussed  in  Section  3.3  and  illustrated  in  Sections 
3.4  and  4.  Related  sampling-based  computational  procedures 
involving  alternatives  to  probability  for  the  representation  of 
epistemic  uncertainty  are  discussed  and  illustrated  in  Ref.  [80], 

To  this  point,  only  the  effects  of  epistemic  uncertainty  on  the 
margin  M(R,  P)  have  been  considered.  Specifically,  M(R,  P)  has 
been  assumed  to  be  a  function  M(R,  P|eM)  of  a  vector  eM  of 
epistemically  uncertain  analysis  inputs.  However,  many  analyses 
involve  an  additional  class  of  uncertainty  known  as  aleatory 
uncertainty  (Section  2).  Specifically,  aleatory  uncertainty  corre¬ 
sponds  to  some  form  of  random  variability  associated  with  the 
system  under  study.  As  examples,  such  variability  might  corre¬ 
spond  to  (i)  variability  in  a  population  of  manufactured  devices, 
(ii)  variability  in  the  effects  of  aging  processes,  (iii)  variability  in 
the  conditions  associated  with  a  particular  class  of  accidents,  or 
(iv)  variability  over  time  in  the  environmental  conditions  that  a 
single  device  or  a  population  of  devices  is  subjected  to.  Alter¬ 
native  descriptors  used  for  aleatory  uncertainty  include  variabil¬ 
ity,  stochastic  uncertainty,  and  irreducible  uncertainty. 

Many  analyses  use  probability  to  represent  both  aleatory 
uncertainty  and  epistemic  uncertainty.  The  mathematics  of  prob¬ 
ability  is  the  same  for  both  uncertainty  representations;  however, 
the  concepts  being  represented  are  very  different.  The  distinction 
between  aleatory  uncertainty  and  epistemic  uncertainty  is 
already  present  in  the  formal  development  of  probability  that 
began  in  the  late  sixteen  hundreds.  The  distinction  between 
aleatory  uncertainty  and  epistemic  uncertainty  is  fundamental 
to  the  design,  implementation  and  interpretation  of  analyses  for 
many  complex  systems. 


If  an  analysis  incorporates  the  effects  of  aleatory  uncertainty, 
then  underlying  the  analysis  there  must  be  a  probability  space 
( A ,  A,  pA)  for  aleatory  uncertainty.  Each  element  a  of  A  is  a  vector 
a  =  [ai,a2,...]  (6.22) 

that  characterizes  one  possible  state  of  the  system.  For  example,  if 
the  failures  in  a  population  of  devices  over  a  specified  time 
interval  [u,  v]  are  under  consideration,  then  a  might  have  the 

a=[n,ti,t2 . t„],  (6.23) 

where  n  is  the  number  of  failed  devices  in  time  interval  [u,  v]  and 
q  is  the  time  of  failure  i  with  u  <  ti  <  t2  <  ■■■  <t„<v. 

A  step  up  in  complexity  is  for  a  to  have  the  form 
a=[n,t1,p1,t2,p2 . tn,pn],  (6.24) 

where  each  pf  is  a  vector  of  random  properties  associated  with  the 
failure  at  time  t,-.  In  an  actual  analysis  for  a  real  system,  the 
definition  of  a,  and  hence  the  associated  probability  space  ( A ,  A, 
pA),  can  be  very  complicated.  Notional  analyses  involving  aleatory 
uncertainty  are  presented  in  Sections  3.6  and  5  and  also  in  Ref. 
[80],  Further,  three  analyses  for  real  systems  involving  aleatory 
uncertainty  are  presented  in  Ref.  [79], 

Similarly  to  the  probability  space  {EM,  EM,  pEM)  for  epistemic 
uncertainty,  the  probability  space  (A,  A,  pA)  for  aleatory  uncer¬ 
tainty  is  usually  defined  by  specifying  conditions  that  define 
distributions  for  the  elements  of  a.  For  example,  the  probability 
space  associated  with  vectors  a  of  the  form  indicated  in  Eq.  (6.23) 
might  be  arrived  at  through  the  assumption  that  device  failures 
are  consistent  with  a  Poisson  process  defined  by  a  rate  X  (yr  T). 
Further,  the  probability  space  associated  with  vectors  a  of  the 
form  indicated  in  Eq.  (6.24)  might  be  arrived  at  through  the 
assumption  again  that  device  failures  are  consistent  with  a 
Poisson  process  defined  by  a  rate  X  and  the  additional  specifica¬ 
tion  of  a  joint  probability  distribution  for  the  elements  of  the 
property  vector  p  conditional  on  the  occurrence  of  a  failure  at  a 
specific  time  t. 

In  practice,  the  probability  space  {A,  A,  pA)  for  aleatory 
uncertainty  is  unlikely  to  be  precisely  known.  Specifically,  many 
of  the  quantities  employed  in  the  definition  of  {A,  A,  pA)  are  likely 
to  be  uncertain  in  an  epistemic  sense.  For  example,  the  occur¬ 
rence  of  a  certain  type  of  event  might  be  assumed  to  follow  a 
Poisson  process  with  a  rate  X  that  is  imprecisely  known.  This  lack 
of  knowledge  about  X  is  epistemic  uncertainty  and  leads  to 
uncertainty  with  respect  to  the  appropriate  values  for  probabil¬ 
ities  that  derive  from  X.  The  very  important  point  being  made 
here  is  that  there  can  be,  and  usually  is,  epistemic  uncertainty 
present  in  the  characterization  of  quantities  used  in  the  definition 
of  the  probability  space  [A,  A,  pA)  for  aleatory  uncertainty.  For 
notational  purposes,  these  quantities  can  be  represented  by  a 
vector  eA  in  analogy  to  the  vectors  eR  and  eP  introduced  in 
conjunction  with  Eq.  (6.16).  With  the  introduction  of  eA,  there  is 
now  a  probability  space  (EA,  EA,  Pea)  that  characterizes  the 
uncertainty  in  eA  and  is  developed  in  a  manner  similar  to  that 
previously  described  for  the  probability  space  (EM,  EM,  pEM) 
associated  with  eM.  For  notational  convenience,  the  effects  of  eA 
on  (A,  A,  pA)  can  be  indicated  by  representing  the  density 
function  associated  with  (A,  A,  pA )  by  dA(a  eA).  In  general,  eA 
can  affect  the  definition  of  the  sample  space  A  but,  for  notational 
simplicity,  this  potential  effect  is  typically  not  indicated. 

The  vector  of  epistemically  uncertain  analysis  inputs  now  has 
the  form 

e  =  [e„,eR,ep]  =  [eA,eM],  (6.25) 

and  the  corresponding  probability  space  (E,  E,  pE)  derives  from  the 
properties  of  the  probability  spaces  (EA,  EA,  Pea)  and  (EM,  EM,  p™)- 
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The  consideration  of  the  uncertainty  in  margins  is  now 
returned  to.  In  concept,  each  possible  realization  a  of  aleatory 
uncertainty  could  lead  to  a  different  performance  of  the  system 
under  consideration.  Notationally,  this  performance  can  be  repre¬ 
sented  by  a  vector 

P„(a|eP)  =  [PA1(a|eP),PA2(a|eP) . PA„0(a|eP)],  (6.26) 

where  PAj-(a|eP),  j=  1,2 . nO,  are  outcomes  of  an  analysis  given 

realization  a  of  aleatory  uncertainty  (i.e„  a  s  A)  and  conditional 
on  realization  eP  of  epistemic  uncertainty  (i.e.,  e=[eA,  eR,  eP]e£). 
As  an  example  from  reactor  risk  assessment,  PA1(a|eP)  could  be 
the  number  of  early  fatalities,  PA2(a|eP)  could  be  the  number  of 
latent  cancer  fatalities,  and  PA3(a|eP)  could  be  the  economic  cost 
for  a  reactor  accident  with  properties  defined  by  the  vector  a  and 
conditional  on  the  values  for  the  epistemically  uncertain  analysis 
inputs  contained  in  eP.  In  the  examples  from  reactor  safety  and 
radioactive  waste  disposal  in  Sections  2  and  3  of  Ref.  [79],  nO 
equals  5  and  2,  respectively. 

In  practice,  performance  measures  used  in  comparisons  with 
requirements  are  likely  to  be  based  on  the  distributions  for  the 
individual  elements  PAj(a|eP)  of  PA(a|eP)  that  derive  from  aleatory 
uncertainty.  In  this  situation,  the  performance  measures  used  in 
comparisons  with  requirements  are  defined  by  a  functional 
relationship  of  the  form 


P[PA(a|eP)|eA]  =  [P1(eA,eP),P2(eA,eP) . P„R(eA,eP)],  (6.27) 

where  the  conditionality  on  eA  in  P[PA(a|eP)|eA]  (i.e.,  “|eA”) 

indicates  that  operations  on  the  elements  PAj(a|eP),  j= 1,2 . 

nO,  of  PA(a|eP)  to  obtain  the  elements  P/eA,eP),  j=  1,2,  ...,  nR,  of 
P[PA(a|eP)|eA]  are  conditional  on  the  probability  space  (A,  A,  pA) 
associated  with  eA. 


Three  examples  of  possible  definitions  for  P,(eA,eP)  follow. 
First,  P,(eA,eP)  might  be  the  expected  value  for  PAj(a|eP)  associated 
with  the  probability  space  {A,  A,  pA)  that  derives  from  eA.  In  this 

Pj(eA,eP)  =  Jj>Aj(  a  |  eP)dA(a  |  eA)dA.  (6.28) 

Second,  Pj(eA,eP )  might  be  the  q  quantile  (e.g.,  q=0.05,  0.5,  0.95)  of 
the  distribution  of  PAj(a|eP)  associated  with  the  probability  space 
(A,  A,  pA)  that  derives  from  eA.  In  this  case,  Pj{eA,eP)  is  the  value  of 
P  such  that 

q  =  Jj>p[PAj(a  |  eP)]dA(a  |  eA)dA.  (6.29) 

Third,  P,(eA,eP)  might  be  the  CCDF  for  PAj<a|eP)  that  derives  from 
the  probability  space  [A,  A,  pA)  associated  with  eA.  In  this  case, 
Pj(eA,eP)  is  a  function  defined  by  the  points 

[p,  Jlp[PAj(a  |  eP)]d„(a  |  eA)dAj ,  (6.30) 

with  problem  specific  knowledge  used  to  limit  the  range  of  P.  The 
expressions  in  Eqs.  (6.28)-(6.30)  may  seem  complicated,  but  expres¬ 
sions  of  this  type  are  routinely  approximated  in  risk  assessments  for 
complex  systems  (e.g.,  see  analyses  in  Ref.  [79]).  In  practice,  the 
definitions  of  the  elements  P/eA,eP)  of  P[PA(a|eP)jeA]  could  be  more 
or  less  complex  than  indicated  in  Eqs.  (6.28)-(6.30). 

The  determination  of  uncertainty  in  margins  is  now  returned 
to.  Once  the  determination  P[PA(a|eP)|eA]  is  completed,  the 
determination  and  representation  of  the  uncertainty  in  margins 
is  the  same  as  previously  discussed  in  conjunction  with 
Eqs.  (6.13)-(6.16).  Specifically,  margin  is  defined  by  a  function 
M(R,  P|e),  which  now  has  the  form 

M(R.P|e)  =  M{R(eR),P[PA(a|eP)|eA]},e  =  [eA,eR,eP],  (6.31) 

when  stated  in  complete  generality.  In  turn,  the  uncertainty  in  the 
margin  M(R,  P|e)  is  defined  as  indicated  in  Eqs.  (6.18)  and  (6.19) 


and,  in  practice,  is  usually  approximated  as  indicated  in 
Eqs.  (6.20)  and  (6.21).  Examples  of  the  uncertainty  in  margin 
results  that  derive  from  aleatory  uncertainty  are  presented  in 
Section  5  and  in  Ref.  [80]  for  notional  analyses  and  in  Ref.  [79]  for 
three  real  analyses  involving  reactor  safety  and  radioactive  waste 
disposal. 

As  noted  in  conjunction  with  Eqs.  (6.7)— (6.1 1 ),  the  meaning 
and  analysis  of  margin  is  more  complex  when  M(R,  P|e)  is  a 
vector  rather  than  a  scalar.  Also,  it  is  anticipated  that  eR  will  not 
be  present  in  most  analyses. 

6.3.  Quantification  in  QMU 

Quantification  in  the  context  of  QMU  is  now  considered.  Such 
quantification  has  two  distinct  and  important  parts.  The  first  part 
is  the  definition  of  the  mathematical  components  that  underlie 
QMU  in  a  particular  analysis.  The  second  part  is  the  actual 
performance  of  the  necessary  calculations  with  these  components 
to  obtain  a  numerical  representation  for  the  uncertainty  asso¬ 
ciated  with  the  margin  or  margins  of  interest.  Two  cases  are 
considered:  (i)  Analyses  involving  only  epistemic  uncertainty,  and 
(ii)  Analyses  involving  both  aleatory  and  epistemic  uncertainty. 

Case  1:  The  case  involving  only  epistemic  uncertainty  is 
considered  first.  For  full  generality,  the  vectors  eR  and  eP  of 
epistemically  uncertain  quantities  are  assumed  to  be  present  in 
the  analysis  under  consideration,  although  this  may  not  be  the 
case  for  a  specific  analysis.  In  particular,  eR  is  likely  to  be  absent 
from  many  analyses,  with  the  result  that  requirements  placed  on 
the  system  would  be  characterized  by  a  single  vector  R  rather 
than  by  a  vector  function  R(eR). 

For  this  case,  the  first  part  of  the  quantification  process  entails 
the  definition  (i.e.,  mathematical  characterization)  of  four  analysis 
components:  (i)  a  function  R(eR)  that  defines  the  requirements 
that  are  to  be  met  conditional  on  realization  eR  of  epistemic 
uncertainty,  (ii)  a  function  P(eP)  that  defines  system  performance 
conditional  on  realization  eP  of  epistemic  uncertainty,  (iii)  a 
probability  space  (EM,  EM ,  pEM)  that  characterizes  the  epistemic 
uncertainty  associated  with  eM=[eR,  eP],  and  (iv)  a  function  M(R, 
P|eM)=M[R(eR),  P(eP)]  that  defines  the  margin  associated  with 
R(eR)  and  P(eP).  In  many  analyses,  R(eR)  and  P(eP)  may  be  one 
dimensional  (i.e.,  scalars);  {EM,  EM,  pEM)  will  probably  be  defined 
by  specifying  distributions  for  the  individual  elements  of  eM; 
and  P(eP)  is  likely  to  be  a  complex  mathematical  structure 
(e.g.,  a  system  of  nonlinear  partial  differential  equations)  that 
requires  a  sophisticated  computer  program  for  evaluation.  If  the 
preceding  quantities  are  not  clearly  defined,  then  the  analysis  is 
inadequately  documented  and,  as  a  consequence,  it  is  difficult 
to  know  what  any  QMU  results  obtained  from  the  analysis 
really  mean. 

The  second  part  of  quantification  in  this  case  involves  carrying 
out  the  calculations  required  to  obtain  an  approximation  to  the 
distribution  of  M[R(eR),  P(eP)]  that  results  from  the  epistemic 
uncertainty  associated  with  eM=  [eR,  eP]  and  characterized  by  the 
probability  space  (EM,  EM,  Pfm).  For  most  analyses,  it  is  antici¬ 
pated  that  a  sampling-based  approach  of  the  form  indicated  in 
conjunction  with  Eqs.  (6.18)— (6.21 )  will  be  used  to  numerically 
approximate  the  distribution  that  characterizes  the  uncertainty  in 
the  margin  M(R,  P|eM)=M[R(eR),  P(eP)].  The  major  computational 
cost  in  this  quantification  will  most  likely  be  the  numerical 
evaluation  of  Pjen)  in  the  sums  indicated  in  Eqs.  (6.20)  and 
(6.21)  as  the  evaluations  of  R(eRi)  and  M[R(eRi),  P(ePi)]  are  unlikely 
to  be  numerically  demanding.  However,  it  is  important  to 
recognize  that  human  cost  rather  than  computational  cost  will 
dominate  the  cost  of  most. 

Case  2:  The  case  involving  both  aleatory  uncertainty  and  epistemic 
uncertainty  is  now  considered.  For  full  generality,  the  vectors  eA,  eR, 
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and  eP  of  epistemically  uncertain  quantities  are  assumed  to  be 
present  in  the  analysis  under  consideration,  although  this  may  not 
be  the  situation  for  a  specific  analysis.  As  for  Case  1,  eR  is  likely  to  be 
absent  from  many  analyses,  with  the  result  that  the  requirements 
placed  on  the  system  would  be  characterized  by  a  single  vector  R 
rather  than  by  a  vector  function  R(eR). 

For  this  case,  the  first  part  of  the  quantification  process  entails 
the  definition  (i.e.,  mathematical  characterization)  of  six  analysis 
components:  (i)  a  function  R(e«)  that  defines  the  requirements 
that  are  to  be  met  conditional  on  realization  eR  of  epistemic 
uncertainty,  (ii)  a  function  P,i(a|eP)  the  defines  system  perfor¬ 
mance  given  realization  a  of  aleatory  uncertainty  and  conditional 
on  realization  eP  of  epistemic  uncertainty  (see  Eq.  (6.26)),  (iii)  a 
probability  space  {A,  A,  pA)  that  characterizes  aleatory  uncer¬ 
tainty  conditional  on  realization  eA  of  epistemic  uncertainty  (see 
Eqs.  (6.22)-(6.24)),  (iv)  a  probability  space  (£,  E,  pE)  that  char¬ 
acterizes  the  epistemic  uncertainty  associated  with  e=  [e^,  eR,  eP] 
(see  Eq.  (6.25)),  (v)  a  function  P[PA(a|eP)|e/i]  that  determines 
summary  measures  of  system  behavior  that  derive  from  aleatory 
uncertainty  for  comparison  with  the  requirements  contained  in 
R(eK)  conditional  on  realizations  eP  and  eA  of  epistemic  uncer¬ 
tainty  (see  Eqs.  (6.27)-(6.30)),  and  (vi)  a  function  M{R(eR), 
P[P,i(a|eP)|eyd}  that  defines  a  margin  based  on  requirement  R(eK) 
and  performance  PlP^alep)^]  conditional  on  realization  e=[eA, 
eR,  eP]  of  epistemic  uncertainty  (see  Eq.  (6.31)). 

The  second  part  of  quantification  for  Case  2  involves  carrying 
out  the  calculations  required  to  obtain  an  approximation  to  the 
distribution  of  M(R,  P|e)=M{R(eR),  P[P„(a|eP)  e^]}  that  results 
from  the  epistemic  uncertainty  associated  with  e=[eA,  eR,  eP] 
and  characterized  by  the  probability  space  (£,  E,  pE).  As  for  Case  1, 
it  is  anticipated  that  most  analyses  will  use  a  sampling-based 
approach  of  the  form  indicated  in  conjunction  with  Eqs.  (6.18)-(6.21) 
to  numerically  approximate  the  distribution  that  characterizes 
the  uncertainty  in  the  margin  M(R,  P|e).  Interior  to  this  calcula¬ 
tion  for  a  given  sample  element  eRi,  ePl]  of  the  form 

indicated  in  conjunction  with  Eqs.  (6.20)  and  (6.21),  it  is  neces¬ 
sary  to  estimate  (i)  R(eRi),  (ii)  P[P/,(a|e„)|e/li],  and  (iii)  M{R(eRi), 
P[PA(a|eP,)|e/i,]}.  The  estimation  or,  most  likely,  exact  determina¬ 
tion  of  the  quantities  in  (i)  and  (iii)  is  anticipated  to  be  straight¬ 
forward  in  most  analyses.  In  contrast,  the  determination  of 
PLP^aleftOle/ti]  could  be  a  major  computational  challenge.  This 
challenge  exists  because  the  determination  of  P[P>i(a | ep,-) | e^,-] 
must  be  preceded  by  an  estimation  of  the  distribution  of  P^alep,) 
conditional  on  the  probability  space  [A,  A,  pA)  for  aleatory 
uncertainty  associated  with  eAi.  In  most  large  analyses,  the  major 
computational  complexity  and  cost  is  associated  with  the  deter¬ 
mination  of  the  distribution  of  P,i(a|ePi)  for  each  pair  [e*,-,  ePi]  of 
sampled  values  for  eP  and  eA.  In  many  large  analyses  (e.g., 
probabilistic  risk  assessments  for  nuclear  power  plants;  see 
Section  2,  Ref.  [79])  extensive  use  is  made  of  fault  trees  and 
event  trees  in  this  determination.  In  addition,  extensive  modeling 
of  physical  processes  is  usually  required. 

As  is  the  case  for  the  example  analyses  in  Ref.  [79],  a  great  deal 
of  careful  planning  and  computational  organization  is  required  to 
successfully  carry  out  an  analysis  for  a  complex  system  that 
involves  a  separation  of  aleatory  and  epistemic  uncertainty. 
However,  without  this  separation,  the  results  of  the  analysis  are 
likely  to  provide  limited  and  possibly  misleading  insights  into  the 
potential  behavior  of  the  system  and  extent  of  our  knowledge 
with  respect  to  this  behavior. 

6.4.  Presentation  of  QMU  results 

Excessive  simplification  in  the  presentation  of  QMU  results 
should  be  avoided.  For  analyses  that  involve  only  epistemic 
uncertainty,  the  best  presentation  format  is  provided  by 


cumulative  or  complementary  cumulative  summaries  of  the  uncer¬ 
tainty  in  analysis  results  of  interest  (i.e„  by  CDFs  or  CCDFs  if 
probability  is  used  to  characterize  epistemic  uncertainty)  with  a 
vertical  line  used  to  indicate  the  specified  requirement  on  system 
performance  (e.g.,  see  Fig.  4.1  and,  more  generally,  Fig.  3.2  in  Ref. 
[80]).  Specifically,  cumulative  summaries  are  appropriate  when 
system  performance  has  a  specified  lower  bound  (see  Fig.  4.1a  and 
also  Fig.  3.2a  in  Ref.  [80]),  and  complementary  cumulative  sum¬ 
maries  are  appropriate  when  system  performance  has  a  specified 
upper  bound  (see  Fig.  4.1b  and  also  Fig.  3.2b  in  Ref.  [80]).  This 
presentation  format  clearly  shows  the  actual  values  for  the 
performance  measure  of  interest,  the  uncertainty  in  this  measure, 
the  specified  bound  on  this  measure,  and  the  implications  of 
uncertainty  with  respect  to  compliance  with  the  specified  bound. 

A  less  informative  presentation  is  provided  by  a  cumulative 
summary  of  the  uncertainty  in  the  margin  for  the  system  perfor¬ 
mance  measure  and  associated  requirement  under  consideration 
(e.g.,  see  Fig.  4.2  and  also  Fig.  3.3  in  Ref.  [80]).  Cumulative 
summaries  are  appropriate  for  margins  because  small  margins 
and,  in  particular,  negative  margins  are  undesirable.  Margin 
summaries  are  less  informative  than  performance  summaries 
because  they  obscure  the  actual  value  of  the  performance  measure 
and  the  relationship  of  this  measure  to  its  associated  requirement. 
Of  course,  given  the  numerical  value  of  the  requirement,  it  is 
mathematically  possible  to  convert  from  margin  values  to  the 
values  for  the  performance  measure.  However,  this  is  not  as  easy  as 
simply  directly  looking  at  the  summary  of  the  actual  values  for  the 
performance  measure  and  its  associated  requirement.  If  it  is 
desired  to  show  margin  results  as  illustrated  in  Fig.  4.2  and  in 
Fig.  3.3  in  Ref.  [80],  it  is  recommended  that  actual  performance 
results  as  illustrated  in  Fig.  4.1  and  in  Fig.  3.2  in  Ref.  [80]  also  be 
shown  as  this  will  help  the  reader  recognize  the  nature  of  the 
performance  results  that  give  rise  to  the  presented  margins. 

When  margins  arise  from  complex  requirements  (e.g.,  upper 
and  lower  bounds  on  performance  over  a  time  interval  as 
illustrated  in  Section  4.3),  the  display  of  the  actual  performance 
is  more  complex  than  simply  showing  a  cumulative  or  comple¬ 
mentary  cumulative  summary  for  a  single  result.  However,  some 
way  of  showing  the  actual  performance  of  the  system  should  be 
sought.  For  the  example  system  and  associated  requirements  in 
Section  4.3,  this  is  accomplished  by  displaying  the  time-depen¬ 
dent  behavior  of  the  system  together  with  the  required  bounds  on 
this  behavior  (Fig.  4.9)  in  addition  to  cumulative  margin  results 
(Fig.  4.10;  also  Fig.  3.8  in  Ref.  [80]).  In  general,  the  exact  form  of 
such  displays  will  be  analysis  specific. 

The  statement  is  often  made  that  the  final  outcome  of  a  QMU 
analysis  should  be  a  summary  measure  of  the  form  “margin/ 
uncertainty”.  For  purposes  of  illustration,  summary  measures  of  this 
form  are  extensively  presented  in  Sections  4  and  5  and  also  in 
Ref.  [79],  However,  these  “margin/uncertainty”  results  provide  a  very 
poor  representation  of  the  outcome  of  a  QMU  analysis  as  too  much 
meaningful  information  is  lost  when  the  results  of  a  complex  analysis 
are  reduced  to  a  single  number.  After  all,  the  fundamental  motivation 
for  performing  an  uncertainty  analysis  derives  from  the  recognition 
that  it  is  not  possible  to  use  a  single  number  to  represent  the  existing 
knowledge  about  the  behavior  of  a  system.  For  example,  the  “margin/ 
uncertainty"  results  in  Eqs.  (4.42)  and  (4.43)  do  not  adequately 
capture  the  more  detailed  results  in  Figs.  4.9  and  4.10  that  they  are 
derived  from  and  are  intended  to  summarize.  This  same  pattern  of 
lost  information  with  “margin/uncertainty”  summaries  for  QMU 
analyses  is  repeated  for  all  examples  presented  in  Sections  4  and  5 
and  also  in  Ref.  [79],  Bluntly  put,  “margin/uncertainty"  results  do  not 
contain  enough  information  to  provide  a  basis  for  appropriately 
informed  decisions  (see  Section  4.5  for  additional  discussion). 

For  analyses  that  involve  aleatory  and  epistemic  uncertainty, 
presentations  of  analysis  results  should  include  displays  that 
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clearly  show  the  separate  effects  of  aleatory  uncertainty  and 
epistemic  uncertainty.  As  an  example,  the  analyses  presented  in 
Sections  5.1  and  4.3  of  Ref.  [80]  involve  aleatory  and  epistemic 
uncertainty,  with  (i)  the  effects  of  aleatory  uncertainty  condi¬ 
tional  on  specific  realizations  of  epistemic  uncertainty  shown  in 
Fig.  5.1a,  (ii)  the  effects  of  epistemic  uncertainty  on  the  perfor¬ 
mance  quantity  of  interest  shown  in  Fig.  5.1b  and  in  Fig.  4.2  of 
Ref.  [80]  and  (iii)  the  effects  of  epistemic  uncertainty  on  margin 
shown  in  Fig.  5.2  and  in  Fig.  4.3  of  Ref.  [80],  This  presentation 
format  provides  a  more  informative  transfer  of  information  than 
the  “margin/uncertainty”  results  shown  in  Eqs.  (5.11)-(5.14)  and 
intended  to  summarize  the  information  contained  in  Fig.  5.1a  and 
b.  For  complex  analyses  involving  aleatory  and  epistemic  uncer¬ 
tainty  of  the  form  illustrated  in  Ref.  [79],  single  “margin/uncer¬ 
tainty”  summaries  simply  cannot  capture  the  information 
provided  by  the  analysis  about  system  behavior  and  the  uncer¬ 
tainty  present  in  our  ability  to  predict  this  behavior. 

In  addition  to  uncertainty  results,  a  QMU  analysis  should  also 
present  sensitivity  analysis  results  [56],  Such  results  play  a  funda¬ 
mental  role  in  analyses  of  complex  systems  by  providing  (i)  insights 
into  system  behavior,  (ii)  guidance  on  how  to  invest  resources  to 
reduce  uncertainty  in  the  assessment  of  system  behavior,  and  (iii)  a 
powerful  tool  for  analysis  verification.  An  uncertainty  analysis 
without  an  associated  sensitivity  analysis  is  incomplete. 

A  fundamental  part  of  the  presentation  of  any  QMU  analysis 
should  be  quality  documentation.  Unfortunately,  many  large 
analyses  are  not  well  documented.  This  is  probably  due  in  part  to 
a  tendency  to  underestimate  the  time  and  resources  required  to 
produce  quality  documentation  for  a  large  analysis.  The  reality  is  that 
it  can  never  be  expected  that  everyone  will  agree  with  the  manner  in 
which  a  large  analysis  is  conducted  and  consequently  with  the 
results  of  that  analysis.  However,  everyone  should  be  able  to  know 
exactly  what  was  assumed  and  done  in  the  analysis.  This  necessary 
knowledge  can  only  result  through  quality  documentation. 
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